| clarkb | jhesketh: but if ** implementation is easy enough then go for it | 00:00 |
|---|---|---|
| jhesketh | right, I just have a fear that we'll miss something in all of hte templating etc that we won't upload a log for a while and it'll be too long before it gets notice | 00:01 |
| jhesketh | d | 00:01 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Revert "Disable voting on pylint because of novaclient breakage" https://review.openstack.org/158498 | 00:03 |
| *** sarob has joined #openstack-infra | 00:04 | |
| clarkb | jhesketh: good point | 00:04 |
| jhesketh | clarkb: it's okay, a recursive search isn't difficult even if I have to walk it by hand | 00:04 |
| *** melwitt has joined #openstack-infra | 00:05 | |
| *** hdd has joined #openstack-infra | 00:06 | |
| *** dims has quit IRC | 00:08 | |
| *** marcusvrn has quit IRC | 00:08 | |
| *** tnovacik has quit IRC | 00:09 | |
| *** bswartz has joined #openstack-infra | 00:10 | |
| fungi | huh... something has changed with the novaclient on puppetmaster... "ERROR (EndpointNotFound): publicURL endpoint for compute service in IAD region not found" | 00:12 |
| fungi | though it's working for hpcloud | 00:12 |
| fungi | i get that error for all the rax regions i try | 00:12 |
| *** baoli has joined #openstack-infra | 00:13 | |
| fungi | novaclient 2.21.0 | 00:13 |
| jeblair | fungi: i wonder/worry if the same thing would happen to nodepool if we restarted it | 00:13 |
| fungi | yeah... | 00:13 |
| jhesketh | clarkb: some searching later... https://github.com/miracle2k/python-glob2/ looks like what we need :-) | 00:14 |
| *** amitgandhinz has joined #openstack-infra | 00:15 | |
| *** sarob has quit IRC | 00:22 | |
| fungi | mordred: you were complaining about this same error in #-dev roughly a month ago, raised by keystoneclient? | 00:22 |
| *** abhirc has quit IRC | 00:25 | |
| *** yamamoto_ has quit IRC | 00:26 | |
| fungi | looks like `keystone catalog` returns nothing at all for compute!?! | 00:27 |
| fungi | i'm not even seeing a cloudServersOpenStack listed in there | 00:29 |
| *** nelsnelson has quit IRC | 00:29 | |
| *** dimtruck is now known as zz_dimtruck | 00:29 | |
| clarkb | fungi: do you have a patch for the grub thing yet? | 00:29 |
| fungi | clarkb: yeah, lemme get you the url | 00:30 |
| clarkb | fungi: I found it, shoudl've looked harder first | 00:30 |
| clarkb | it failed a puppet job | 00:30 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: zuul-swift-upload support recursive globs https://review.openstack.org/158514 | 00:30 |
| jhesketh | clarkb: ^ | 00:30 |
| clarkb | fungi: looks like centos6 puppet failed due to the git timeout | 00:31 |
| clarkb | fungi: I will rechek | 00:31 |
| fungi | clarkb: thanks, and yeah, i found that and rechecked already | 00:31 |
| clarkb | oh you got it | 00:31 |
| *** jhesketh has quit IRC | 00:31 | |
| *** jhesketh has joined #openstack-infra | 00:32 | |
| fungi | clarkb: so any guesses why there's no compute or server service in rackspace's catalog at all now? http://paste.openstack.org/show/180871/ | 00:32 |
| *** marcusvrn has joined #openstack-infra | 00:32 | |
| fungi | am i doing a stupid? | 00:32 |
| clarkb | fungi: I don't think so | 00:33 |
| clarkb | fungi: that looks like the right way to list things | 00:33 |
| fungi | pretty sure that's why novaclient has started complaining that it can't find a service url | 00:33 |
| clarkb | ya I would agree | 00:33 |
| clarkb | and itisn't that the tenant id changed, the service isn't listed at all | 00:33 |
| clarkb | so are we 100% hpcloud right now? | 00:34 |
| fungi | maybe https://identity.api.rackspacecloud.com/v2.0/ is no longer what we're supposed to be hitting for that? | 00:34 |
| clarkb | oh that could be | 00:34 |
| clarkb | endpoint moved or they switched to v3 | 00:34 |
| fungi | yeah, headed down that route now | 00:34 |
| mtreinish | fungi: fwiw, on my personal rax account keystone catalog returns nothing, but nova list still works | 00:36 |
| mtreinish | using the same v2.0 auth url | 00:36 |
| fungi | mtreinish: okay, that helps, thanks | 00:37 |
| fungi | mtreinish: novaclient 2.21.0? | 00:37 |
| fungi | and keystoneclient 1.1.0? | 00:37 |
| mtreinish | fungi: keystoneclient 1.0.0 and novaclient 2.20.0 | 00:37 |
| mtreinish | looks like 1 release behind you for both | 00:38 |
| jroll | clarkb: fungi: the endpoint should still work | 00:39 |
| jroll | we didn't move it | 00:39 |
| fungi | yeah, and even with the versions mtreinish is using i still get "publicURL endpoint for compute service in DFW region not found" | 00:40 |
| mtreinish | fungi: both releases are newish (nova last week, keystone a couple weeks ago) | 00:40 |
| fungi | so something is not quite right in this environment. i'll try from one of my machines | 00:40 |
| mtreinish | fungi: oh, I have OS_REGION_NAME=iad, I'll try dfw | 00:40 |
| fungi | mtreinish: well, i get it for iad and ord too | 00:41 |
| *** erlon has quit IRC | 00:41 | |
| mtreinish | yeah, no issues for me with any of the regions | 00:41 |
| fungi | actually, i take that back. with keystoneclient 1.0.0 and novaclient 2.20.0 i just get ERROR (SSLError): [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed | 00:42 |
| *** dims has joined #openstack-infra | 00:42 | |
| *** mtanino has joined #openstack-infra | 00:42 | |
| fungi | maybe they've updated their endpoint server cert and the chain is insufficient? | 00:42 |
| *** asselin_ has joined #openstack-infra | 00:43 | |
| jroll | fungi: you have the rackspace novaclient plugin installed? | 00:43 |
| jogo | On the other hand, the strongest argument for some people advocating | 00:44 |
| jogo | 4.0 seems to have been a wish to see 4.1.15 - because "that was the | 00:44 |
| jogo | version of Linux skynet used for the T-800 terminator | 00:44 |
| jogo | amazing | 00:44 |
| fungi | jroll: no, using the standard novaclient support, no rax extension | 00:44 |
| jroll | fungi: I don't think that works, afaik :x | 00:44 |
| * jroll tries | 00:44 | |
| fungi | jroll: it _was_ working for us | 00:44 |
| jroll | orly | 00:44 |
| jroll | hrm | 00:44 |
| jroll | we haven't changed anything afaik | 00:44 |
| *** tmcpeak has quit IRC | 00:45 | |
| jogo | https://lkml.org/lkml/2015/2/22/203 (source) | 00:45 |
| fungi | and yeah, the cert validity is Not Before: Nov 14 2011, Not After : Nov 12 2016 so i doubt the cert has changed recently | 00:45 |
| *** amitgandhinz has quit IRC | 00:47 | |
| *** marun has joined #openstack-infra | 00:47 | |
| *** asselin_ has quit IRC | 00:48 | |
| openstackgerrit | Merged openstack-infra/infra-manual: It's openstack-admins in launchpad https://review.openstack.org/155973 | 00:48 |
| fungi | OKAY! rolling back novaclient, keystoneclient _and_ requests fixes this | 00:48 |
| adam_g | clarkb, hey, popping online for a few, saw this: https://review.openstack.org/#/c/158247/ | 00:48 |
| *** Ryan_Lane has quit IRC | 00:49 | |
| fungi | now to see if i can trim this down to a more minimal reproduction | 00:49 |
| clarkb | adam_g: did that actually fix it? | 00:49 |
| adam_g | clarkb, not sure | 00:49 |
| clarkb | I am pulling up the log now | 00:49 |
| clarkb | oh hrm nothing to sync on that change to ironic | 00:50 |
| clarkb | need to rebase the neutronclient bump if thats not done yet | 00:50 |
| clarkb | or no need for rebase since the fix merged | 00:50 |
| adam_g | clarkb, http://logs.openstack.org/46/158246/1/check/check-requirements-integration-dsvm/994aba9/console.html | 00:51 |
| adam_g | yea | 00:51 |
| adam_g | looks good | 00:51 |
| adam_g | well, still breaking on that other requirement sync issue | 00:51 |
| *** zul has quit IRC | 00:51 | |
| clarkb | adam_g: I think we need to make sure 157606 does the ironic sync properly | 00:51 |
| adam_g | addressed by https://review.openstack.org/#/c/158086/ (fixing it now) | 00:52 |
| *** zul has joined #openstack-infra | 00:52 | |
| adam_g | clarkb, we can squash everything together into one patch once the stack looks okay | 00:52 |
| clarkb | adam_g: I have rechecked it | 00:52 |
| clarkb | adam_g: where it is 157606 | 00:52 |
| arosen | clarkb: I've got my nodepool node now working again though I'm still seeing this error periodically: http://logs2.aaronorosen.com/24/157624/1/check/dsvm-tempest-full-congress-pg-nodepool/6be371b/logs/devstacklog.txt.gz#_2015-02-24_00_51_16_112 | 00:52 |
| fungi | okay, keystoneclient has nothing to do with this. have to roll novaclient back to 2.20.0 and requests back to 2.5.1 to get things working again | 00:52 |
| arosen | I'm curious if this is just me or if there is anything I can do to help avoid this? | 00:53 |
| arosen | ReadTimeoutError: HTTPSConnectionPool(host='pypi.python.org', port=443): Read timed out. | 00:53 |
| arosen | Perhaps I should try and setup a local pypi cache ? | 00:53 |
| *** yamamoto has joined #openstack-infra | 00:53 | |
| openstackgerrit | Adam Gandelman proposed openstack/requirements: Do not sync requirements to possibly capped library's master branch https://review.openstack.org/158086 | 00:55 |
| fungi | ftr, `pip install -U requests==2.5.1 python-novaclient==2.20.0` results in a working environment but neither `pip install -U python-novaclient` nor `pip install -U python-novaclient==2.20.0` work | 00:55 |
| *** zz_dimtruck is now known as dimtruck | 00:56 | |
| *** ChuckC has quit IRC | 00:56 | |
| *** dannywil_ has quit IRC | 00:57 | |
| openstackgerrit | Adam Gandelman proposed openstack/requirements: Do not sync requirements to possibly capped library's master branch https://review.openstack.org/158086 | 00:58 |
| fungi | jroll: this is with no novaclient plugin and setting OS_USERNAME, OS_PASSWORD, OS_TENANT_NAME, OS_PROJECT_ID, OS_VERSION=2, OS_SERVICE_NAME=cloudServersOpenStack and OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/ | 00:58 |
| jroll | fungi: hrm. | 00:59 |
| jroll | fungi: so requests > 2.5.1 will break it? | 00:59 |
| fungi | jroll: seems that way | 00:59 |
| * jroll tries | 00:59 | |
| fungi | jroll: and novaclient 2.21.0 | 00:59 |
| fungi | either one seems to | 01:00 |
| *** MaxV has joined #openstack-infra | 01:00 | |
| *** ZZelle_ has quit IRC | 01:00 | |
| fungi | oh, also had OS_REGION_NAME set obviously | 01:00 |
| adam_g | clarkb, the stack ending at https://review.openstack.org/#/c/158276/ should hopefully pass, then we can squash it down into 157606 (assuming the integration.sh change is good for master) | 01:02 |
| clarkb | adam_g: kk | 01:02 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add a comment for ERROR_ON_CLONE https://review.openstack.org/158417 | 01:04 |
| *** MaxV has quit IRC | 01:05 | |
| jroll | fungi: what keystoneclient version? | 01:05 |
| jroll | I can't get auth through :x | 01:05 |
| fungi | jroll: latest (1.1.0) or previous (1.0.0) both worked for me | 01:06 |
| *** MaxV has joined #openstack-infra | 01:06 | |
| jroll | huh | 01:06 |
| fungi | jroll: unset NOVA_RAX_AUTH and use your tenant password rather than a separate api key | 01:06 |
| jroll | ah, password, ok | 01:06 |
| *** baoli has quit IRC | 01:07 | |
| fungi | jroll: my understanding was that the rackspace novaclient plugin was something to do with api key support | 01:07 |
| fungi | but novaclient seems to support password auth to rackspace just fine without it | 01:07 |
| jroll | right, I always forget | 01:07 |
| fungi | i wonder if this issue has something to do with the urllib3 import changes in requests 2.5.2 | 01:08 |
| *** ayoung has joined #openstack-infra | 01:09 | |
| jroll | possibly | 01:09 |
| jroll | or they're verifying differently | 01:09 |
| *** MaxV has quit IRC | 01:09 | |
| jroll | SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) | 01:09 |
| jroll | maybe that's an error now | 01:09 |
| *** adam_g is now known as adam_g_out | 01:09 | |
| *** MaxV has joined #openstack-infra | 01:09 | |
| jroll | that issue is still open, though | 01:09 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:10 | |
| fungi | jroll: yeah, and https://github.com/rackspace/pyrax/issues/528 is related | 01:10 |
| fungi | apparently it's surfacing there as well for similar reasons | 01:11 |
| *** devlaps has quit IRC | 01:11 | |
| jroll | fungi: reasons like, our cert is old and crusty | 01:11 |
| fungi | indeedie | 01:11 |
| fungi | er, indeedily-doodily | 01:12 |
| jroll | fwiw, I bugged people about this like, the second day it started happening :( | 01:12 |
| *** melwitt has quit IRC | 01:14 | |
| *** ddieterly has joined #openstack-infra | 01:15 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/infra-specs: Spec to refactor puppet code in openstack_project https://review.openstack.org/137471 | 01:15 |
| openstackgerrit | Merged openstack-infra/infra-specs: Add migrate askbot to infra spec https://review.openstack.org/154061 | 01:16 |
| *** MaxV has quit IRC | 01:18 | |
| *** wenlock has quit IRC | 01:18 | |
| *** david-lyle is now known as david-lyle_afk | 01:19 | |
| *** MaxV has joined #openstack-infra | 01:19 | |
| *** baoli has joined #openstack-infra | 01:19 | |
| *** rfolco has quit IRC | 01:19 | |
| *** MaxV has quit IRC | 01:20 | |
| *** rfolco has joined #openstack-infra | 01:20 | |
| *** MaxV has joined #openstack-infra | 01:20 | |
| *** achanda has quit IRC | 01:21 | |
| *** radez_g0n3 is now known as radez | 01:22 | |
| *** rkukura has joined #openstack-infra | 01:23 | |
| *** hdd has quit IRC | 01:24 | |
| *** otter768 has joined #openstack-infra | 01:24 | |
| fungi | 69d9594 seems to be the last working commit for requests. b86330d "Update certificate bundle." appears to introduce the failure | 01:27 |
| *** MaxV has quit IRC | 01:27 | |
| *** MaxV has joined #openstack-infra | 01:28 | |
| fungi | so probable cause for the requests-related issues is the cert bundle in requests 2.5.2 is unable to validate the rackspace api endpoint | 01:28 |
| fungi | sigmavirus24_awa: ^ if you're not actually "_awa" | 01:28 |
| *** otter768 has quit IRC | 01:28 | |
| fungi | going to see now if i can similarly bisect what happened between python-novaclient 2.20.0 and 2.21.0 | 01:29 |
| *** MaxV has quit IRC | 01:31 | |
| *** MaxV has joined #openstack-infra | 01:31 | |
| dimtruck | fungi: you're awesome :) | 01:32 |
| fungi | dimtruck: it's a double-bisection. what better way to spend a monday night? | 01:33 |
| dimtruck | that was our issue in solum too! Started ~2.30 pm. We had too many people out to look into it but we also thought it was with requests | 01:33 |
| dimtruck | hahaha, yes!! | 01:33 |
| *** MaxV_ has joined #openstack-infra | 01:33 | |
| jroll | fungi: ouch | 01:33 |
| * jroll wonders where in the cert chain this is failing | 01:33 | |
| fungi | jroll: i haven't dug deeper on that front, mainly because i'd need to spend a few minutes with the requests docs | 01:34 |
| jroll | fungi: indeed, I don't expect you to | 01:34 |
| jroll | I wonder where that bundle comes from as well | 01:34 |
| jroll | (in requests) | 01:34 |
| jroll | it's all through thawte | 01:35 |
| *** hdd has joined #openstack-infra | 01:35 | |
| *** MaxV__ has joined #openstack-infra | 01:35 | |
| *** ParsectiX has quit IRC | 01:35 | |
| *** MaxV has quit IRC | 01:36 | |
| *** ParsectiX has joined #openstack-infra | 01:36 | |
| *** mbacchi has quit IRC | 01:36 | |
| *** Ryan_Lane has joined #openstack-infra | 01:36 | |
| fungi | jroll: originated from this pr https://github.com/kennethreitz/requests/pull/2442 | 01:36 |
| fungi | "produced by mkcert.org" | 01:36 |
| jroll | fungi: ya, removes thawte | 01:37 |
| jroll | ouch | 01:37 |
| fungi | oh, cool! | 01:37 |
| *** bdpayne has quit IRC | 01:37 | |
| *** MaxV_ has quit IRC | 01:37 | |
| *** MaxV has joined #openstack-infra | 01:37 | |
| f1ller | looks like thawte was dropped @ https://github.com/kennethreitz/requests/commit/b86330d1f15f04e77093800efb8c6bd6909e8c8e | 01:37 |
| jroll | Once a day, mkcert.org obtains the latest list of Mozilla's trusted root CA certificates. It parses the file, removes anything explicitly marked as untrusted, and then keeps the data in memory. | 01:37 |
| fungi | yep | 01:38 |
| fungi | fun times | 01:38 |
| *** MaxV has quit IRC | 01:38 | |
| *** MaxV has joined #openstack-infra | 01:39 | |
| *** MaxV__ has quit IRC | 01:39 | |
| *** abhirc has joined #openstack-infra | 01:40 | |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/devstack-gate: Configure sub-nodes for Neutron DVR aiopcpu https://review.openstack.org/158525 | 01:41 |
| jroll | strangely, this has data: curl https://mkcert.org/generate/ | grep -i thawte | 01:41 |
| f1ller | looks like thawte is still there ... https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1 :D | 01:41 |
| *** MaxV has quit IRC | 01:42 | |
| jroll | so wth | 01:42 |
| *** otter768 has joined #openstack-infra | 01:42 | |
| fungi | yeah, but maybe on the 10th there was a problem | 01:42 |
| *** MaxV has joined #openstack-infra | 01:42 | |
| fungi | since that's when the pr says the update was generated | 01:42 |
| f1ller | http://bonsai.mozilla.org/cvslog.cgi?file=mozilla/security/nss/lib/ckfw/builtins/certdata.txt&rev=HEAD&mark=1.87 | 01:42 |
| f1ller | nah | 01:42 |
| jroll | yeah | 01:43 |
| *** krtaylor has quit IRC | 01:43 | |
| fungi | also could have just been an incomplete pr | 01:43 |
| *** MaxV has quit IRC | 01:43 | |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/devstack-gate: Configure sub-nodes for Neutron DVR aiopcpu https://review.openstack.org/158525 | 01:44 |
| *** MaxV has joined #openstack-infra | 01:44 | |
| *** MaxV has quit IRC | 01:45 | |
| f1ller | actually, they dont rock the cvs anymore ;) | 01:45 |
| f1ller | https://hg.mozilla.org/releases/mozilla-release/log/a2ffa9047bf4/security/nss/lib/ckfw/builtins/certdata.txt | 01:46 |
| *** jamielennox is now known as jamielennox|away | 01:46 | |
| *** MaxV has joined #openstack-infra | 01:46 | |
| *** koolhead17 has joined #openstack-infra | 01:46 | |
| *** koolhead17 has joined #openstack-infra | 01:46 | |
| *** abhirc has quit IRC | 01:48 | |
| *** koolhead17 has quit IRC | 01:48 | |
| jroll | could have been a mkcert.org issue as well | 01:48 |
| *** asselin_ has joined #openstack-infra | 01:48 | |
| f1ller | yeah, hope they log stdout+stderr on those cron jobs :P | 01:48 |
| *** tiswanso has joined #openstack-infra | 01:49 | |
| *** abhirc has joined #openstack-infra | 01:49 | |
| *** MaxV has quit IRC | 01:49 | |
| *** asselin_ has quit IRC | 01:49 | |
| *** MaxV has joined #openstack-infra | 01:50 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 01:50 | |
| fungi | okay, for this evening's second bisect, python-novaclient 99a4439 (the corresponding merge commit for https://review.openstack.org/105900 "Support using the Keystone V3 API from the Nova CLI") has broken this | 01:50 |
| fungi | the merge commit prior to that works for me (when paired with a working requests version anyway) | 01:51 |
| *** amitgandhinz has joined #openstack-infra | 01:51 | |
| jroll | fungi: what's the failure mode for that part? | 01:52 |
| fungi | jroll: ERROR (EndpointNotFound): publicURL endpoint for compute service in IAD region not found | 01:52 |
| jroll | ah, ok | 01:52 |
| *** MaxV has quit IRC | 01:53 | |
| fungi | morganfainberg: if you're around, any guesses what about that ^ change could have brought that about? | 01:53 |
| fungi | i pick on you purely because you're listed as co-author ;) | 01:53 |
| *** MaxV has joined #openstack-infra | 01:53 | |
| *** tiswanso_ has joined #openstack-infra | 01:53 | |
| jroll | I assume that does something that our identity system doesn't handle correctly | 01:54 |
| fungi | i was just having trouble tracking it down because of being coupled with an entirely independent issue with requests | 01:54 |
| *** koolhead17 has joined #openstack-infra | 01:54 | |
| *** koolhead17 has joined #openstack-infra | 01:54 | |
| fungi | jroll: yeah, my guess would be it works with keystone. just not with something almost but not quite entirely unlike keystone ;) | 01:55 |
| jroll | yep :( | 01:55 |
| morganfainberg | fungi, hi | 01:55 |
| morganfainberg | fungi, whats up? | 01:55 |
| *** adalbas has quit IRC | 01:55 | |
| *** gyee has quit IRC | 01:55 | |
| *** krtaylor has joined #openstack-infra | 01:55 | |
| fungi | morganfainberg: https://review.openstack.org/105900 on novaclient (keystone v3 support) seems to cause it to no longer be able to quthenticate to rackspace | 01:55 |
| fungi | authenticate too, but quthenticate is more fun | 01:55 |
| morganfainberg | jamielennox|away, ping | 01:56 |
| morganfainberg | uhm. | 01:56 |
| morganfainberg | hmm. | 01:56 |
| *** amitgandhinz has quit IRC | 01:56 | |
| morganfainberg | because rax doesn't use keystone >.> | 01:56 |
| *** tiswanso has quit IRC | 01:56 | |
| fungi | morganfainberg: yep | 01:56 |
| jroll | I wonder if our roles stuff is screwy | 01:56 |
| morganfainberg | i mean.. that doesn't answer it clearly | 01:56 |
| jroll | or something, idk | 01:56 |
| fungi | morganfainberg: no surprise, their novaclient auth plug-in doesn't exhibit this issue, but novaclient prior to that change was doing regular password auth to them just dandy | 01:57 |
| morganfainberg | hmmmmmm. | 01:57 |
| fungi | at least according to my bisecting | 01:57 |
| morganfainberg | so i'd need to see what the difference between their auth plugin and the normal username/password is. | 01:58 |
| morganfainberg | i'm going to guess it's something like... rax doesn't do v3 api at all. | 01:58 |
| morganfainberg | and expects some extra data value that we don't pass [or barfs on some extra data value we do pass] | 01:58 |
| fungi | morganfainberg: https://github.com/rackerlabs/rackspace-auth-openstack | 01:58 |
| *** MaxV has quit IRC | 01:58 | |
| jroll | morganfainberg: indeed we don't do v3 api | 01:58 |
| morganfainberg | aha | 01:58 |
| morganfainberg | thnx | 01:58 |
| *** MaxV has joined #openstack-infra | 01:59 | |
| morganfainberg | is this all uses of client or just CLI? | 01:59 |
| morganfainberg | and does openstack client work? | 01:59 |
| morganfainberg | jroll, ^ | 01:59 |
| fungi | morganfainberg: good question--i'm only experiencing it with the cli | 01:59 |
| fungi | but that's also all i've been testing | 01:59 |
| jroll | morganfainberg: I'm not sure, but actually a teammate of mine was having client problems earlier | 01:59 |
| morganfainberg | fungi, i would assume it would be *really* broken for things like zuul/nodepool if it wasn't working in the raw client | 02:00 |
| * jroll reads scrollback | 02:00 | |
| fungi | morganfainberg: for nodepool anyway, and yes we're afraid to restart nodepoold now :/ | 02:00 |
| *** MaxV has quit IRC | 02:00 | |
| morganfainberg | fungi, and things would be exploding left and right. | 02:00 |
| jroll | bleh, that was unrelated | 02:00 |
| morganfainberg | fungi, can we spin up a test nodepool that will do 1 instance build? | 02:00 |
| morganfainberg | fungi, or similar [i don't have a rax account so...] | 02:00 |
| *** MaxV has joined #openstack-infra | 02:01 | |
| morganfainberg | fungi, or is that a lot of work [tell me it is, and i wont aks you to] | 02:01 |
| fungi | morganfainberg: well, i won't be because i'm about to crash for the night and am leaving on a long vacation tomorrow, but yes it should be possible to test similar codepaths anyway | 02:01 |
| morganfainberg | fungi, oh vacayshun?! is that allowed ;) | 02:01 |
| morganfainberg | fungi, have a good one!! | 02:02 |
| fungi | it's possible nodepool-dev is working set up to hit rackspace. checking now | 02:02 |
| *** MaxV has quit IRC | 02:02 | |
| fungi | oh, right, nodepool-dev is not a thing | 02:02 |
| fungi | that was just the replacement nodepool server dns name | 02:02 |
| * fungi sighs | 02:02 | |
| *** abhirc has quit IRC | 02:02 | |
| morganfainberg | fungi, no worries | 02:03 |
| morganfainberg | let me work through this with jroll | 02:03 |
| jroll | fungi: this is working for me for novaclient==2.21.0 | 02:03 |
| morganfainberg | jroll, now i'm sad i closed my rax account, testing would be easier if i wasn't worried about being charged. | 02:03 |
| fungi | jroll: were you able to reproduce our working auth settings? e.g. not using rax auth plugin? | 02:03 |
| morganfainberg | jroll, does that client include that v3 patch fix? | 02:03 |
| *** MaxV has joined #openstack-infra | 02:04 | |
| *** bhuvan has quit IRC | 02:04 | |
| *** MaxV has quit IRC | 02:04 | |
| fungi | morganfainberg: yeah, that was the commit between 2.20.0 and 2.21.0 where it stops working for me | 02:04 |
| jroll | morganfainberg: I think that's where the bug originated, was that version | 02:04 |
| morganfainberg | ok | 02:04 |
| jroll | fungi: yes, I'm not using rax plugin or anything | 02:04 |
| *** MaxV has joined #openstack-infra | 02:04 | |
| fungi | jroll: how very interesting. what envvars do you set? | 02:04 |
| *** signed8bit_ZZZzz is now known as signed8bit | 02:04 | |
| fungi | maybe i have a stray envvar that's causing this bug to surface | 02:05 |
| morganfainberg | fungi, which is also something we should make sure is fixed ;) | 02:05 |
| jroll | OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/ OS_REGION_NAME=IAD NOVA_SERVICE_NAME=cloudServersOpenStack OS_PASSWORD=blah OS_USERNAME=blah OS_TENANT_NAME=blah OS_VERSION=2 OS_PROJECT_ID=blah nova list | 02:05 |
| morganfainberg | fungi, if it's something we can catch. | 02:05 |
| jroll | fungi: ^ | 02:05 |
| fungi | agreed | 02:05 |
| morganfainberg | jroll, OMG I KNOW YOUR PASSWORD NOW! IT's BLAH! | 02:05 |
| fungi | jroll: thanks, giving just those a try | 02:05 |
| *** MaxV has quit IRC | 02:05 | |
| jroll | morganfainberg: ya, but that's a fake username ;) | 02:05 |
| morganfainberg | jroll, ;) | 02:05 |
| morganfainberg | Oh hunter2? | 02:06 |
| morganfainberg | ;) | 02:06 |
| jroll | ******* | 02:06 |
| morganfainberg | exactly | 02:06 |
| *** MaxV has joined #openstack-infra | 02:06 | |
| *** bhuvan has joined #openstack-infra | 02:06 | |
| jroll | so idk if this is a bug | 02:07 |
| *** MaxV has quit IRC | 02:07 | |
| morganfainberg | happy to help fix it if it really is a bug | 02:07 |
| morganfainberg | if it's not... i dunno what i can do to fix it ;) | 02:07 |
| jroll | I can yell at rackspace keystone people, that's about all I've got :| | 02:08 |
| jroll | and I'm not in a position where they would likely listen to me | 02:08 |
| morganfainberg | yell at them to really use keystone ;) | 02:08 |
| *** MaxV has joined #openstack-infra | 02:08 | |
| fungi | okay! so that seems to work. which suggests one of the additional envvars we're setting is triggering this. now to narrow down which one | 02:08 |
| jogo | fungi: as a 'TP Person' what does that mean? | 02:08 |
| jroll | morganfainberg: supposedly that's in the works | 02:08 |
| jogo | fungi: http://www.openstack.org/legal/transparency-policy/ | 02:08 |
| morganfainberg | jroll, yes. | 02:08 |
| morganfainberg | jroll, i think the non-persistent token thing is the big blocker atm | 02:09 |
| morganfainberg | jroll, but that is close. | 02:09 |
| *** otter768 has quit IRC | 02:09 | |
| jroll | idk what that is | 02:09 |
| morganfainberg | jroll, not needing to store every token in a SQL table ;) | 02:09 |
| morganfainberg | jroll, it doesn't scale well... | 02:09 |
| jroll | heh | 02:09 |
| jroll | yeah | 02:09 |
| * jroll remembers back to finding 10mm tokens in a django table | 02:09 | |
| morganfainberg | ahaha ouch | 02:10 |
| openstackgerrit | Aaron Rosen proposed openstack-infra/project-config: Gate on gate-python-congressclient-python34 https://review.openstack.org/158531 | 02:10 |
| morganfainberg | ooh endpoint not found. | 02:10 |
| morganfainberg | fungi, silly question, were you not asking for a scoped token [e.g. left off the tenant]? | 02:11 |
| *** MaxV has quit IRC | 02:11 | |
| morganfainberg | fungi, because that endpoint not found error is indicative of unscoped tokens (not sure if you can get an unscoped token from RAX) | 02:11 |
| jroll | yeah, I'm wondering if it has to do with roles/scopes | 02:11 |
| fungi | jroll: morganfainberg: !!! OS_SERVICE_NAME=cloudServersOpenStack has ceased working with that commit | 02:11 |
| jroll | uh | 02:11 |
| jroll | I'm using that | 02:11 |
| jroll | OH | 02:11 |
| *** MaxV has joined #openstack-infra | 02:11 | |
| jroll | no, I'm using NOVA_SERVICE_NAME | 02:12 |
| jroll | bwahaha | 02:12 |
| morganfainberg | oh wait. your account has the old name. | 02:12 |
| morganfainberg | oooooooh | 02:12 |
| fungi | yeah | 02:12 |
| jroll | WELP | 02:12 |
| morganfainberg | this was mordred's filtering issue | 02:12 |
| morganfainberg | with shade | 02:12 |
| fungi | when i add NOVA_SERVICE_NAME it works fine | 02:12 |
| morganfainberg | you're getting the old endpoint name thing in your catalog | 02:12 |
| fungi | so, regression? | 02:12 |
| morganfainberg | sortof. | 02:12 |
| *** jamielennox|away is now known as jamielennox | 02:12 | |
| fungi | or just rax problem | 02:12 |
| jroll | I mean | 02:13 |
| morganfainberg | i mean.. it's rax giving you bad data in the catalog and we actually are caring about it somewhat now | 02:13 |
| jroll | no longer supporting a config optino seems like a regression | 02:13 |
| *** MaxV has quit IRC | 02:13 | |
| *** marun has quit IRC | 02:13 | |
| fungi | and yeah, i found you and mordred discussing it with the keystone client lib in #-dev about a month ago | 02:13 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set custom cpu_model for live_migrate https://review.openstack.org/141530 | 02:13 |
| morganfainberg | so partially a regression? | 02:13 |
| * morganfainberg says that with squinty-eyes | 02:13 | |
| *** MaxV has joined #openstack-infra | 02:13 | |
| *** dims has quit IRC | 02:14 | |
| *** asettle is now known as asettle-luncheon | 02:14 | |
| fungi | welp, anyway, i'm going to do a quick scour for anywhere we're using the other var i guess | 02:14 |
| morganfainberg | fungi, iirc it's because the account your using has data that it can't figure out | 02:14 |
| morganfainberg | fungi, most everyone else wont ever see that issue | 02:14 |
| morganfainberg | jroll, can you use the other option? | 02:14 |
| jroll | morganfainberg: ya, this didn't break me, I was just trying to help | 02:15 |
| morganfainberg | jroll i mean it works w/ the env var fungi was using for you? | 02:15 |
| *** MaxV_ has joined #openstack-infra | 02:15 | |
| morganfainberg | or is it similarly broken when not using the NOVA_bnlah option? | 02:15 |
| jroll | oh,s ec | 02:15 |
| jroll | morganfainberg: wfm | 02:16 |
| morganfainberg | jroll, yeah it's this account has the old service in it too | 02:16 |
| *** signed8bit has quit IRC | 02:16 | |
| jroll | which seems strange | 02:16 |
| morganfainberg | jroll, so any account that doesn't have the old service attached to it wont get it in the catalog | 02:16 |
| fungi | with no NOVA_SERVICE_NAME and OS_SERVICE_NAME=cloudServersOpenStack instead? | 02:16 |
| jroll | fungi: correct, I can use OS_SERVICE_NAME | 02:17 |
| fungi | aha, "old service" in this case being a second compute service | 02:17 |
| fungi | the "rackspace legacy servers" or whatever | 02:17 |
| morganfainberg | yep | 02:17 |
| jroll | huh | 02:17 |
| fungi | e.g. pre-nova | 02:17 |
| jroll | how do I get a service catalog | 02:17 |
| fungi | keystone catalog | 02:17 |
| morganfainberg | jroll, it's magic *waves hands* | 02:17 |
| jroll | oh wow, we have a compute service | 02:18 |
| morganfainberg | uhm, it's part of the token body | 02:18 |
| morganfainberg | in all seriousness | 02:18 |
| jroll | rather than a cloudServersOpenStack service | 02:18 |
| morganfainberg | yeah. | 02:18 |
| morganfainberg | and nova hard-codes compute | 02:18 |
| jroll | fungi: so I take it your catalog has a cloudServersOpenStack service and not a compute service? | 02:18 |
| morganfainberg | jroll, he has two compute services | 02:18 |
| morganfainberg | jroll, iirc | 02:18 |
| jroll | fungi: do y'all have any first gen servers? | 02:18 |
| jroll | ah | 02:19 |
| *** MaxV has quit IRC | 02:19 | |
| morganfainberg | hte pre-nova stuff, that doesn't work anymore.. but it's still in the catalog | 02:19 |
| morganfainberg | i *think* thats it | 02:19 |
| fungi | jroll: my catalog actually shows no compute services at all, which i think is maybe a display issue | 02:19 |
| jroll | huh | 02:19 |
| fungi | jroll: and yeah we still have a handful of first-gen servers in at least some of our tenants, can't recall if this particular one does | 02:19 |
| jroll | morganfainberg: yeah, first gen | 02:19 |
| jroll | ok | 02:19 |
| morganfainberg | fungi, yeah mordred came to the conclusion that almost no one would ever see this because they wouldn't have first gen too | 02:20 |
| * jroll cries at the fact that catalogs vary due to this | 02:20 | |
| *** MaxV_ has quit IRC | 02:20 | |
| fungi | morganfainberg: jroll: okay, thanks! mystery solved, at least | 02:20 |
| morganfainberg | jroll, i just argued with someone we shouldn't be putting other random data in catalogs | 02:20 |
| fungi | and this is far less egregious than today's requests release dropping thawte entirely from their cert bundle | 02:20 |
| jroll | morganfainberg: indeed | 02:20 |
| *** MaxV has joined #openstack-infra | 02:21 | |
| morganfainberg | jroll, to be honest i think that deployers are putting $random_thing$ in the catalog that isn't openstack is asking for problems | 02:21 |
| jroll | fungi: ya, do you want me to put up a PR for that or? | 02:21 |
| morganfainberg | they should not be relying on openstack to provide a catalog if you don't use openstack for... using it. | 02:21 |
| fungi | jroll: go for it. i haven't even gotten around to opening an issue yet. maybe someone else already has | 02:21 |
| jroll | morganfainberg: on the other hand, how do you make wannabe-openstack services discoverable | 02:21 |
| jroll | oh, you answered that | 02:21 |
| * jroll ponders | 02:21 | |
| jroll | fungi: sure, I'll poke around | 02:21 |
| fungi | thanks again | 02:22 |
| *** SumitNaiksatam has quit IRC | 02:22 | |
| *** MaxV has quit IRC | 02:22 | |
| morganfainberg | jroll, i actually would like to offer an option in the catalog that can be better defined as not-openstack or a target for another catalog service that isn't saying "yep you can use this stuff but good luck figuring out how to auth with it w/o apriori knowledge" | 02:22 |
| *** MaxV has joined #openstack-infra | 02:23 | |
| morganfainberg | fungi, so.. we can probably work on something to fix this, but i'd need to ask rax to give me a similarly wierd acocunt [or have you guys to a ton of testing for me] | 02:23 |
| morganfainberg | fungi, i'm happy to do so. | 02:23 |
| morganfainberg | fungi, but not sure how easy this will be to chase down. | 02:23 |
| jroll | morganfainberg: yeah, it's an interesting problem, especially when some of those want to be openstack services (see also rax:database being our trove service) | 02:23 |
| morganfainberg | for a real fix that is. | 02:23 |
| fungi | morganfainberg: nah, i'm just adding NOVA_SERVICE_NAME everywhere it's an issue | 02:24 |
| morganfainberg | fungi, okie | 02:24 |
| *** MaxV has quit IRC | 02:24 | |
| *** abhirc has joined #openstack-infra | 02:24 | |
| morganfainberg | fungi, let me know if you need me to do anything here. i'll happily do so | 02:24 |
| *** MaxV has joined #openstack-infra | 02:24 | |
| morganfainberg | fungi, or if you get roadblocked anywhere else that you can actually blame on something keystone did/does/not-quite-keystone-does | 02:24 |
| fungi | appreciate the offer. hopefully this is the last we'll have to worry about it though | 02:24 |
| morganfainberg | fungi, until next novaclient release *duck* | 02:25 |
| fungi | hah | 02:25 |
| *** marun has joined #openstack-infra | 02:25 | |
| *** markvoelker has quit IRC | 02:25 | |
| morganfainberg | jroll, well does the token work with rax:database? | 02:25 |
| morganfainberg | jroll, if that is the case, i'd say it's fine [this is the definition], and has a clear endpoint. e.g. my authorization will currently work with it | 02:25 |
| *** markvoelker has joined #openstack-infra | 02:26 | |
| morganfainberg | jroll, without lots of extra hoops. and it can conform to the normal endpoint model of data | 02:26 |
| morganfainberg | jroll, i know repose makes that easier across your entire environment | 02:26 |
| jroll | morganfainberg: but at one point it was "non-openstack stuff in the catalog" | 02:27 |
| jroll | idk. | 02:27 |
| jroll | fungi: https://github.com/kennethreitz/requests/issues/2456 | 02:27 |
| morganfainberg | jroll, right. | 02:27 |
| fungi | jroll: awesome | 02:27 |
| morganfainberg | jroll, which is where i'd like to find some better construct to store that in | 02:27 |
| jamielennox | morganfainberg: sorry - i'm here | 02:27 |
| morganfainberg | jroll, something to make the end users not rip our their hair | 02:27 |
| morganfainberg | jroll, but that is a bigger issue | 02:27 |
| jroll | morganfainberg: oh, I see what you mean, you're saying tokens should work with anything in the catalog -- I agree | 02:28 |
| morganfainberg | jamielennox, we solved it. turns out fungi had multiple compute services | 02:28 |
| jamielennox | morganfainberg: sweet, so not a ksc issue? | 02:28 |
| morganfainberg | well... sortof, i think we have a minor regression here | 02:28 |
| jamielennox | that would almost be a first | 02:28 |
| morganfainberg | but one that is highly specific and hard to fix | 02:28 |
| morganfainberg | because we'd need a wierdly setup RAX account to be 100% sure we got it. | 02:29 |
| jamielennox | multiple of the same service_type in the same catalog? | 02:29 |
| morganfainberg | it's that the infra account has the legacy pre-nova compute service | 02:29 |
| jroll | I mean... we can make that happen | 02:29 |
| morganfainberg | jroll, i'm tempted to ask for an account similarly setup because this seems like someone else might hit it | 02:29 |
| jroll | we'd just add the service and crank first gen quota to 0 | 02:29 |
| morganfainberg | jroll, ooor we just publish a doc that says "yeah don't use <old style>" | 02:29 |
| morganfainberg | thing | 02:30 |
| jroll | morganfainberg: ya, would be nice not to break existing users, though | 02:30 |
| *** markvoelker has quit IRC | 02:30 | |
| morganfainberg | jroll, hm. let me hit you tomorrow on that, lets let it stew overnight see how much we both feel it's needed | 02:30 |
| jroll | assuming they haven't gone through the same thing we just did and figured it out | 02:30 |
| jroll | morganfainberg: cool, I think people that could do this are likely gone for the day anyway | 02:30 |
| morganfainberg | jroll, we might just want to see if we can duplicate it and see how likely anyone will be to hit it. | 02:30 |
| jroll | +1 | 02:30 |
| morganfainberg | jroll, if it's really a tiny tiny % and we can document use X env var to fix | 02:31 |
| morganfainberg | i think thats ok. but if its more wide spread we should chase this down | 02:31 |
| jroll | right... maybe just wait a day and see if a bug is filed :P | 02:31 |
| morganfainberg | jroll, so tomorrow afternoon work for you? | 02:31 |
| jroll | though they'd be likely to blame rackspace, not keystone | 02:31 |
| morganfainberg | post x-project meeting. | 02:31 |
| jroll | sure, I'll be aroudn | 02:31 |
| jroll | perfect | 02:31 |
| morganfainberg | jroll, hah sure blame rax ;) *ducks* keystone is perfect in every way | 02:32 |
| morganfainberg | ;) | 02:32 |
| jroll | :P | 02:32 |
| jroll | I just tend to say all software sucks and everything is to blame | 02:32 |
| jroll | alright, y'all have a good night, I'm out | 02:33 |
| *** MaxV has quit IRC | 02:33 | |
| *** MaxV has joined #openstack-infra | 02:33 | |
| *** abhirc has quit IRC | 02:34 | |
| *** carl_baldwin has quit IRC | 02:34 | |
| *** MaxV has quit IRC | 02:35 | |
| fungi | jeblair: mordred: clarkb: in case you didn't catch any of that, for rackspace apparently setting NOVA_SERVICE_NAME (instead of or in addition to OS_SERVICE_NAME) works around this in latest novaclient and i've updated the shell snippets on puppetmaster accordingly. someone probably still needs to double-check nodepool to see if any similar adjustments are warranted | 02:35 |
| *** MaxV has joined #openstack-infra | 02:35 | |
| *** ryanpetrello has quit IRC | 02:35 | |
| *** ryanpetrello has joined #openstack-infra | 02:36 | |
| *** vilobhmm1 has quit IRC | 02:36 | |
| fungi | jeblair: mordred: clarkb: sigmavirus24_awa: also today's requests release (2.5.2) dropped thawte from their cert bundle and so if it gets upgraded we won't be able to authenticate to rackspace's api endpoint. https://github.com/kennethreitz/requests/issues/2456 | 02:36 |
| fungi | and with that, i'm going to go pack and have some sake | 02:38 |
| fungi | night, all | 02:38 |
| *** MaxV has quit IRC | 02:38 | |
| clarkb | good night | 02:38 |
| clarkb | thanks for update. I am about to talk at a user group about jjb | 02:39 |
| *** MaxV_ has joined #openstack-infra | 02:41 | |
| clarkb | I have a jenkins running on my laptop, we will do this live | 02:41 |
| *** Ryan_Lane has quit IRC | 02:42 | |
| *** yjiang5 has left #openstack-infra | 02:45 | |
| *** mestery has joined #openstack-infra | 02:48 | |
| *** MaxV_ has quit IRC | 02:49 | |
| *** MaxV has joined #openstack-infra | 02:50 | |
| *** MaxV has quit IRC | 02:55 | |
| *** MaxV has joined #openstack-infra | 02:55 | |
| *** nelsnelson has joined #openstack-infra | 02:55 | |
| *** carl_baldwin has joined #openstack-infra | 02:56 | |
| *** MaxV has quit IRC | 02:56 | |
| *** armax has quit IRC | 02:57 | |
| *** MaxV has joined #openstack-infra | 02:57 | |
| *** MaxV has quit IRC | 02:58 | |
| *** MaxV has joined #openstack-infra | 02:59 | |
| *** MaxV has quit IRC | 03:00 | |
| *** MaxV has joined #openstack-infra | 03:01 | |
| *** sarob has joined #openstack-infra | 03:01 | |
| *** unicell1 has quit IRC | 03:02 | |
| *** MaxV has quit IRC | 03:02 | |
| *** MaxV has joined #openstack-infra | 03:02 | |
| *** yamahata has quit IRC | 03:06 | |
| *** markvoelker has joined #openstack-infra | 03:08 | |
| *** MaxV has quit IRC | 03:13 | |
| *** MaxV has joined #openstack-infra | 03:13 | |
| *** marcusvrn has quit IRC | 03:15 | |
| *** MaxV_ has joined #openstack-infra | 03:15 | |
| *** ivar-laz_ has joined #openstack-infra | 03:16 | |
| *** MaxV has quit IRC | 03:17 | |
| *** carl_baldwin has quit IRC | 03:18 | |
| *** rfolco has quit IRC | 03:19 | |
| *** ivar-lazzaro has quit IRC | 03:20 | |
| *** ivar-laz_ has quit IRC | 03:21 | |
| pleia2 | clarkb: nice, good luck on the presentation :) | 03:21 |
| *** Sukhdev has joined #openstack-infra | 03:21 | |
| *** armax has joined #openstack-infra | 03:22 | |
| *** ddieterly has quit IRC | 03:23 | |
| nibalizer | clarkb: is killin it | 03:23 |
| nibalizer | er i mean he's rocking it | 03:23 |
| *** carl_baldwin has joined #openstack-infra | 03:23 | |
| *** MaxV_ has quit IRC | 03:25 | |
| *** armax has quit IRC | 03:26 | |
| *** MaxV has joined #openstack-infra | 03:26 | |
| *** MaxV has quit IRC | 03:27 | |
| pleia2 | :D | 03:27 |
| *** patrickeast has quit IRC | 03:27 | |
| *** dmellado has quit IRC | 03:28 | |
| *** MaxV has joined #openstack-infra | 03:28 | |
| *** dmellado has joined #openstack-infra | 03:29 | |
| *** marun has quit IRC | 03:34 | |
| *** nelsnelson has quit IRC | 03:34 | |
| *** sarob has quit IRC | 03:34 | |
| *** MaxV has quit IRC | 03:35 | |
| *** carl_baldwin has quit IRC | 03:37 | |
| *** sabeen has joined #openstack-infra | 03:37 | |
| *** amitgandhinz has joined #openstack-infra | 03:40 | |
| *** oomichi has quit IRC | 03:41 | |
| *** amitgandhinz has quit IRC | 03:45 | |
| *** armax has joined #openstack-infra | 03:50 | |
| *** asettle-luncheon is now known as asettle | 03:53 | |
| *** dannywilson has joined #openstack-infra | 03:53 | |
| *** stevemar has quit IRC | 03:55 | |
| *** amotoki has joined #openstack-infra | 03:55 | |
| *** stevemar has joined #openstack-infra | 03:55 | |
| *** claudiub has quit IRC | 03:56 | |
| *** dannywilson has quit IRC | 03:57 | |
| *** armax has quit IRC | 04:02 | |
| *** hdd has quit IRC | 04:02 | |
| *** armax has joined #openstack-infra | 04:03 | |
| *** Sukhdev has quit IRC | 04:04 | |
| *** vilobhmm1 has joined #openstack-infra | 04:04 | |
| *** Ryan_Lane has joined #openstack-infra | 04:06 | |
| *** erikmwilson has quit IRC | 04:07 | |
| *** erikmwil- has joined #openstack-infra | 04:07 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remvoe unnecessary warning message. https://review.openstack.org/158543 | 04:07 |
| *** nelsnelson has joined #openstack-infra | 04:09 | |
| *** otter768 has joined #openstack-infra | 04:10 | |
| *** erikmwil- is now known as erikmwilson | 04:11 | |
| *** harlowja is now known as harlowja_away | 04:14 | |
| *** otter768 has quit IRC | 04:14 | |
| *** harlowja_away is now known as harlowja | 04:15 | |
| *** devlaps has joined #openstack-infra | 04:17 | |
| *** armax has quit IRC | 04:17 | |
| *** dkliban is now known as dkliban_afk | 04:19 | |
| *** ddieterly has joined #openstack-infra | 04:23 | |
| *** armax has joined #openstack-infra | 04:23 | |
| *** radez is now known as radez_g0n3 | 04:24 | |
| *** ddieterly has quit IRC | 04:29 | |
| *** bhuvan has quit IRC | 04:30 | |
| *** spzala has quit IRC | 04:31 | |
| *** garyh has quit IRC | 04:36 | |
| jhesketh | clarkb: is git.o.o using a load balancer of sorts? | 04:39 |
| *** markvoelker has quit IRC | 04:39 | |
| * jhesketh can see the haproxy being installed, but can't tell where the A records point to | 04:39 | |
| clarkb | jhesketh: it is, git.o.o is a dns round robin of two haproxy nodes | 04:39 |
| *** markvoelker has joined #openstack-infra | 04:39 | |
| clarkb | jhesketh: git-fe01 and git-fe02 | 04:39 |
| clarkb | we use a round robin because one node doen't have enough outbound internet bandwidth | 04:40 |
| clarkb | jhesketh: then git01/02/03/04/05 run apache and git-daemon to do the serving | 04:40 |
| clarkb | jhesketh: the backends ^ are all directly accessible over ports 8080, 4443, and 29418 as ewll | 04:40 |
| jhesketh | clarkb: okay thanks. I'm trying to figure why I get so many connection failures with our 3rd party stuff | 04:41 |
| jhesketh | is there any rate limiting? | 04:41 |
| clarkb | jhesketh: not beyond what rax does for us | 04:41 |
| clarkb | jhesketh: which we shouldn't hit anymore since the second node | 04:41 |
| jhesketh | clarkb: I can't see any puppet stuff for git-fe, is that the rackspace load balancing product (ie not a node infra manages) | 04:42 |
| clarkb | jhesketh: no, ew didn't use that because support for not http was a but rudimentary for our needs | 04:42 |
| clarkb | jhesketh: let me find the puppet | 04:42 |
| jhesketh | heh, thanks :-) | 04:42 |
| *** ayoung has quit IRC | 04:43 | |
| clarkb | jhesketh: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/git.pp thats the frontend | 04:43 |
| clarkb | jhesketh: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/git_backend.pp thats the backed that runs apache/git-daemon/cgit | 04:43 |
| *** markvoelker has quit IRC | 04:44 | |
| * jhesketh is blind and mis-read the regex | 04:44 | |
| jhesketh | thanks :-) | 04:44 |
| clarkb | np | 04:44 |
| jhesketh | clarkb: I don't know much about haproxy, but do you suppose either haproxy or the puppet module itself has rate limiting? | 04:45 |
| clarkb | jhesketh: now that I think about it yes haproxy does have rate limiting to an extent but its not source based its global | 04:45 |
| clarkb | also apache defaults do too | 04:45 |
| jhesketh | hmm | 04:47 |
| clarkb | jhesketh: I would be surprised if you hitthose though as we seem to hit the bandwidth limiting in rax first | 04:48 |
| clarkb | jhesketh: I won't get to it tonight but if you give me some source IPs I can grep through logs | 04:48 |
| jhesketh | clarkb: yeah I'd be surprised too | 04:48 |
| jhesketh | oh that'd be really handy thanks | 04:48 |
| jhesketh | let me know when you've got time | 04:48 |
| clarkb | ok tomorrow will probably work | 04:49 |
| jhesketh | it's probably a pebcak | 04:49 |
| jhesketh | but something is hurting turbo-hipster | 04:49 |
| *** nelsnelson has quit IRC | 04:49 | |
| jhesketh | and I know some people who are cloning off github because it's been more reliable than git.o.o (which surprised me given the gate experience) | 04:50 |
| clarkb | hrm, ya more info would be good | 04:50 |
| clarkb | in theory we hit these servers harder than anyone else so we should notice if it breaks first | 04:51 |
| clarkb | ok time for me to drop off, but put IPs somewhere and I can look at haproxy infos | 04:51 |
| jhesketh | yeah that's what I would have thought | 04:55 |
| *** ParsectiX has quit IRC | 04:55 | |
| jhesketh | okay cheers, have a good night :-) | 04:55 |
| *** ParsectiX has joined #openstack-infra | 04:55 | |
| *** sputnik13 has quit IRC | 04:55 | |
| *** krtaylor has quit IRC | 04:55 | |
| *** ParsectiX has quit IRC | 04:57 | |
| *** ParsectiX has joined #openstack-infra | 04:58 | |
| *** armax has quit IRC | 05:00 | |
| *** dimtruck is now known as zz_dimtruck | 05:05 | |
| *** krtaylor has joined #openstack-infra | 05:07 | |
| *** armax has joined #openstack-infra | 05:07 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:09 | |
| *** SumitNaiksatam has quit IRC | 05:10 | |
| *** markvoelker has joined #openstack-infra | 05:10 | |
| *** sabeen has quit IRC | 05:13 | |
| *** krtaylor has quit IRC | 05:14 | |
| *** koolhead17 has quit IRC | 05:15 | |
| *** yfried_ has joined #openstack-infra | 05:15 | |
| *** bhunter71 has quit IRC | 05:16 | |
| *** devlaps has quit IRC | 05:17 | |
| *** markvoelker has quit IRC | 05:17 | |
| *** MaxV has joined #openstack-infra | 05:19 | |
| SpamapS | https://tracker.debian.org/pkg/presentty | 05:19 |
| SpamapS | it lives | 05:19 |
| SpamapS | oh my | 05:19 |
| SpamapS | cowsay has an RFA | 05:19 |
| * SpamapS may just have to offer that A | 05:20 | |
| StevenK | Haha | 05:20 |
| StevenK | I keep wanting to find tractorgen | 05:20 |
| SpamapS | cowsay is apparently abandonware | 05:22 |
| *** hdd has joined #openstack-infra | 05:23 | |
| *** MaxV has quit IRC | 05:23 | |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/devstack-gate: Configure sub-nodes for Neutron DVR aiopcpu https://review.openstack.org/158525 | 05:24 |
| *** ddieterly has joined #openstack-infra | 05:25 | |
| *** krtaylor has joined #openstack-infra | 05:26 | |
| *** baoli has quit IRC | 05:27 | |
| SpamapS | tho somebody reimplemented it in R | 05:28 |
| clarkb | does it speak like a pirate? | 05:28 |
| SpamapS | yes, and they distribute it by gathering all the files and packing them in a ... | 05:29 |
| *** ddieterly has quit IRC | 05:29 | |
| * SpamapS is disappointed nobody bit | 05:29 | |
| StevenK | Gah | 05:30 |
| StevenK | pyopenssl is not packaged | 05:30 |
| StevenK | And os-cloud-config needs it | 05:30 |
| *** BharatK has joined #openstack-infra | 05:30 | |
| SpamapS | StevenK: That seems unlikely. | 05:31 |
| SpamapS | StevenK: it's called python-openssl | 05:31 |
| SpamapS | and it's even in Ubuntu main. ;) | 05:32 |
| StevenK | Bah | 05:32 |
| StevenK | python:Depends put it in as python-pyopenssl | 05:32 |
| *** jamielennox is now known as jamielennox|away | 05:33 | |
| SpamapS | StevenK: I believe that means python-openssl is missing an X-something header that explains the mapping. | 05:33 |
| StevenK | SpamapS: Given it's called pyopenssl on pypi, I was a little tripped up | 05:33 |
| *** jamielennox|away is now known as jamielennox | 05:34 | |
| *** achanda has joined #openstack-infra | 05:34 | |
| *** arosen1 has joined #openstack-infra | 05:34 | |
| arosen1 | Hi, I upgraded my jenkins and now it seems like the zmq plugin isn't binding on 8888, any ideas? | 05:35 |
| arosen1 | I've googled around but haven't found anything yet. | 05:35 |
| SpamapS | StevenK: yeah that's weird | 05:35 |
| clarkb | arosen1 it wont bind until a job runs | 05:36 |
| clarkb | also it should log if it tries to bind | 05:36 |
| *** garyh has joined #openstack-infra | 05:37 | |
| arosen1 | clarkb: isn't this what nodepool uses to register nodes with jenkins/ | 05:39 |
| clarkb | yes, but nodepool should register nodes if you have jobs to run | 05:40 |
| clarkb | also early binding would be fine if someone wants to make that change it was easier to bind lazily | 05:41 |
| *** tiswanso has joined #openstack-infra | 05:44 | |
| arosen1 | clarkb: will nodepool only connect to jenkins if it has nodes in ready state? | 05:45 |
| arosen1 | oh sorry nvm jenkins is what binds. | 05:45 |
| clarkb | zmq wont happen until a job starts but nodepool should connect as soon as that happens | 05:45 |
| clarkb | connect retries a frequent | 05:46 |
| clarkb | are* | 05:46 |
| *** tiswanso_ has quit IRC | 05:47 | |
| *** ParsectiX has quit IRC | 05:47 | |
| *** ParsectiX has joined #openstack-infra | 05:47 | |
| arosen1 | clarkb: how do you get a job to run in jenkins? Don't you need nodes registered to run a job? | 05:47 |
| *** deepakcs has joined #openstack-infra | 05:48 | |
| arosen1 | clarkb: ah i rechecked congress on the openstack-dev/sandbox and that got it to bind :) | 05:48 |
| clarkb | yes but you should get nodes without zmq set up | 05:49 |
| clarkb | either due to min ready or via gearman demand | 05:49 |
| *** abhishek has joined #openstack-infra | 05:52 | |
| deepakcs | clarkb, fungi had agreed to help us get ssh access to the rax VM for debuggign the gluster job's oom issue | 05:52 |
| arosen1 | clarkb: ah so zuul talks to jenkins via zeromq. What does nodepool use zmq for if it uses gearman to register nodes? | 05:52 |
| deepakcs | clarkb, 'guess fungi is on vacation, so can u helps us get access to one of the rax VMs ? | 05:52 |
| clarkb | arosen1 the zmq publisher lists job info from the jenkins so it can know when to delete nodes | 05:52 |
| *** tiswanso has quit IRC | 05:52 | |
| clarkb | deepakcs probably but not tonight its late for me | 05:53 |
| arosen1 | clarkb: ah so zuul also talks to nodepool | 05:53 |
| arosen1 | gotcha that's how that min thing works | 05:53 |
| arosen1 | sorry makes sense :) | 05:53 |
| clarkb | arosen ya jenkins to zuul is gearman | 05:53 |
| clarkb | nodepool to zuul is gearman | 05:53 |
| *** koolhead17 has joined #openstack-infra | 05:53 | |
| clarkb | and nodepool to jenkins is zmq | 05:53 |
| arosen1 | cool thanks! Great to know and confirm that | 05:54 |
| deepakcs | clarkb, ok, it could help us with some debug if u could give us today. I am fine with tomorrow tho' | 05:55 |
| deepakcs | clarkb, i would need some instruction on how to kick off the run, so that we are doing the same that fungi does. | 05:55 |
| clarkb | just run what the jenkins job runs its all in the jenkins vonfig | 05:55 |
| deepakcs | clarkb, so the standard pretty_tox.sh ..... cmdline or need to setup some stuff before doing that ? jenkins console log show a lot of stuff happening before pretty_tox runs | 05:56 |
| deepakcs | clarkb, locally we either use pretty_tox or use testr direclty | 05:57 |
| abhishek | hi all, i am using asselin guide for setting up CI environment but after installing master when I am trying to connect to Gearman plugin the connection is failing can anyone help me with that?/ | 05:57 |
| *** coolsvap_ is now known as coolsvap | 05:57 | |
| *** ildikov has quit IRC | 06:00 | |
| clarkb | deepakcs https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1803 is the shell the job runs after https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n326 | 06:06 |
| clarkb | deepakcs you should run that locally to see if it reproduces | 06:06 |
| deepakcs | clarkb, ok, will do, but appreciate if we get access the rax vm where its reproducible consistently. | 06:07 |
| *** erikmwilson has quit IRC | 06:07 | |
| *** erikmwilson has joined #openstack-infra | 06:07 | |
| *** erikmwil- has joined #openstack-infra | 06:08 | |
| *** hdd has quit IRC | 06:08 | |
| clarkb | yes we can do tgat if you cant reproducing using ^ | 06:08 |
| clarkb | abhishek check that geard is running and listening on port 4730 | 06:09 |
| clarkb | zuul starts it by default | 06:09 |
| *** yfried_ has quit IRC | 06:10 | |
| *** otter768 has joined #openstack-infra | 06:11 | |
| deepakcs | clarkb, ok | 06:11 |
| *** harlowja is now known as harlowja_away | 06:11 | |
| abhishek | clarkb : I restarted zuul and zuul merger its working good now :) | 06:12 |
| *** erikmwilson has quit IRC | 06:12 | |
| *** erikmwil- is now known as erikmwilson | 06:12 | |
| *** hdd has joined #openstack-infra | 06:13 | |
| *** markvoelker has joined #openstack-infra | 06:13 | |
| *** doude has quit IRC | 06:13 | |
| *** ParsectiX has quit IRC | 06:13 | |
| *** andreykurilin_ has joined #openstack-infra | 06:14 | |
| *** otter768 has quit IRC | 06:15 | |
| *** tnovacik has joined #openstack-infra | 06:15 | |
| *** markvoelker has quit IRC | 06:17 | |
| *** abhishek has quit IRC | 06:18 | |
| *** sdake__ has joined #openstack-infra | 06:23 | |
| *** sputnik13 has joined #openstack-infra | 06:24 | |
| *** MarkAtwood has joined #openstack-infra | 06:25 | |
| *** ddieterly has joined #openstack-infra | 06:26 | |
| *** sputnik13 has quit IRC | 06:26 | |
| BharatK | clarkb, for the above links you gave to deepakcs we need jenkins setup locally right? | 06:26 |
| *** sdake_ has quit IRC | 06:26 | |
| clarkb | BharatK no its just shell, you can runit whereever | 06:26 |
| BharatK | clarkb, okay | 06:27 |
| *** ddieterly has quit IRC | 06:30 | |
| *** sputnik13 has joined #openstack-infra | 06:34 | |
| *** sarob has joined #openstack-infra | 06:35 | |
| *** pcrews has quit IRC | 06:39 | |
| *** sarob has quit IRC | 06:39 | |
| *** BharatK has quit IRC | 06:40 | |
| *** BharatK has joined #openstack-infra | 06:41 | |
| *** andreykurilin_ has quit IRC | 06:44 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jeepyb: Enforce manage_projects to be run with specified user https://review.openstack.org/152085 | 06:45 |
| *** reed has quit IRC | 06:45 | |
| *** ildikov has joined #openstack-infra | 06:46 | |
| clarkb | yolanda re ^ we should consider similar for nodepool dib builds | 06:46 |
| yolanda | clarkb, nodepool dib doesn't use root? | 06:46 |
| *** pcrews has joined #openstack-infra | 06:46 | |
| *** mrunge has joined #openstack-infra | 06:47 | |
| clarkb | no because the daemon runs as the nodepool user when run manually it needs to run as nodepool user too | 06:48 |
| clarkb | been using sudo -H -u nodepool which works you just have to remember to do that | 06:49 |
| yolanda | ah, so not really talking about the dib image process, but the command line tools, right? | 06:49 |
| yolanda | i created the jeepyb patch because i became exhausted of people entering into the instance and running as root, and all the folder perms messed :) | 06:49 |
| clarkb | yup the cli commands | 06:50 |
| clarkb | dib has the same perms issues in the dib cache | 06:50 |
| yolanda | ok, sounds good to me, yes, same problem | 06:50 |
| yolanda | i'll try to find a hole to work on it | 06:50 |
| clarkb | cool no rush sudo works too :) | 06:50 |
| yolanda | it's mostly to preven disaster | 06:51 |
| yolanda | people don't enter into our nodepool instance so much, but it was a repeating problem with jeepyb | 06:51 |
| *** sputnik13 has quit IRC | 06:52 | |
| yolanda | clarkb, so now that you are around...we've been having a problem with jeepyb. We created a set of projects with track-upstream flag. By circumstances, we had to remove that flag. And now all the projects seem to have problems with git, they keep saying the git project is empty | 06:54 |
| *** prmtl has joined #openstack-infra | 06:54 | |
| yolanda | have you experienced that issue? | 06:54 |
| clarkb | our git project is empty issues are related to ansible | 06:55 |
| *** viglesias has quit IRC | 06:55 | |
| clarkb | when ansible runs puppet it does git mirrors first then gerrit but each uses current master of puppet repos wheb they run | 06:55 |
| clarkb | so if a changes merges at the correct time gerrit is updated but nothing else and the git mirrors have an empty project | 06:56 |
| clarkb | but gerrit itself is fine | 06:56 |
| *** sdake_ has joined #openstack-infra | 06:58 | |
| *** viglesias has joined #openstack-infra | 06:59 | |
| *** sdake__ has quit IRC | 07:01 | |
| *** jcoufal has joined #openstack-infra | 07:03 | |
| *** sputnik13 has joined #openstack-infra | 07:05 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add pylint jobs for networking-ofagent https://review.openstack.org/158582 | 07:05 |
| stevemar | whats the easiest way to see if any project is using a specific file from oslo-incubator? | 07:05 |
| stevemar | preferably without having to clone all projects and performing a grep :) | 07:06 |
| SpamapS | stevemar: google for 'site:git.openstack.org foo.py' ? | 07:06 |
| AJaeger | stevemar, is it enough to check openstack-common.conf for this? | 07:07 |
| stevemar | AJaeger, maybe... i think so | 07:07 |
| *** MaxV has joined #openstack-infra | 07:07 | |
| AJaeger | You can download the file from http://git.openstack.org/organization/project/openstack-common.conf | 07:07 |
| *** sputnik13 has quit IRC | 07:08 | |
| AJaeger | Or in general, would just a wget work for e.g. http://git.openstack.org/ORGANIZATION/PROJECT/PROJECT/openstack/... ? | 07:08 |
| AJaeger | Then you just need a loop over all projects - get it from project-config repository | 07:09 |
| stevemar | AJaeger, okay, let me try those out, i think that'll work | 07:09 |
| stevemar | AJaeger, thank you sir! you always have the answers :) | 07:09 |
| AJaeger | Just stepping in for fungi who has all the answers ;) | 07:10 |
| *** MaxV has quit IRC | 07:12 | |
| stevemar | only 2 places actually use oslo-incubator/fileutils.py :( | 07:12 |
| *** sputnik13 has joined #openstack-infra | 07:13 | |
| AJaeger | REally? MY local find shows manila, zaqar, sahara, trove, designate, neutron, ironic, barbican, nova, horizon, heat, cinder, django_openstack_auth, glance, keystone | 07:14 |
| AJaeger | And that'S not everything checked out - only openstack namespace | 07:14 |
| stevemar | AJaeger, hmm... i searched for "module=fileutils" | 07:14 |
| stevemar | so the openstack-common.conf is pulled up | 07:14 |
| stevemar | ugly google link coming: https://www.google.ca/search?es_sm=122&q=site%3Agit.openstack.org+%22module%3Dfileutils%22&oq=site%3Agit.openstack.org+%22module%3Dfileutils%22&gs_l=serp.3...11922.11922.0.12256.1.1.0.0.0.0.82.82.1.1.0.msedr...0...1c.1.62.serp..1.0.0.bJTE2yD27hQ | 07:15 |
| AJaeger | http://paste.opensuse.org/35283736 | 07:16 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/devstack-gate: Configure sub-nodes for Neutron DVR aiopcpu https://review.openstack.org/158525 | 07:16 |
| AJaeger | so, don't trust google ;( | 07:16 |
| stevemar | AJaeger, yeah, i changed my search now, looks like quite a few projects don't have it in their .conf? | 07:16 |
| stevemar | blah, silly google | 07:16 |
| AJaeger | stevemar, see my paste above, and compare openstack-common.conf and find | 07:17 |
| stevemar | okay so people *are* using it | 07:17 |
| AJaeger | sahara is one project that does not have it in openstack-common.conf but still has it in the tree | 07:17 |
| AJaeger | same with heat. | 07:17 |
| stevemar | tsk tsk | 07:17 |
| AJaeger | But you have to test yourself, my system has not all repos checked out... | 07:18 |
| stevemar | AJaeger, i was just trying to get a feel for which functions are actually used | 07:18 |
| yolanda | i think gerrit from our side is wrong , clarkb, i'll double check | 07:18 |
| AJaeger | stevebaker, for django_openstack_auth, fileutils comes in via django_openstack_auth/openstack_auth/openstack/common/policy.py | 07:18 |
| *** xyang1 has quit IRC | 07:19 | |
| AJaeger | So, keep in mind, it might be used indirectly | 07:19 |
| AJaeger | same for glance | 07:19 |
| AJaeger | policy.py drags it in | 07:19 |
| *** yfried_ has joined #openstack-infra | 07:22 | |
| *** ddieterly has joined #openstack-infra | 07:26 | |
| *** yfried__ has joined #openstack-infra | 07:27 | |
| *** yfried_ has quit IRC | 07:27 | |
| stevemar | AJaeger, yep, it's the only piece left that oslo.policy drags in: https://github.com/openstack/oslo.policy | 07:27 |
| stevemar | which is why i started to look into it | 07:27 |
| AJaeger | stevemar, I see... | 07:28 |
| stevemar | AJaeger, i was thinking of adding fileutils.py to oslo.utils, but i think maybe creating a new oslo.io might be better... | 07:28 |
| AJaeger | can't help with that decision | 07:28 |
| *** ddieterly has quit IRC | 07:31 | |
| stevemar | AJaeger, i think i'll wait til vancouver to chat about it - there's no rush :) | 07:31 |
| AJaeger | ;) | 07:31 |
| *** jcoufal_ has joined #openstack-infra | 07:32 | |
| *** chlong has quit IRC | 07:34 | |
| *** jcoufal has quit IRC | 07:35 | |
| *** coolsvap is now known as coolsvap_ | 07:36 | |
| *** belmoreira has joined #openstack-infra | 07:38 | |
| *** vilobhmm1 has quit IRC | 07:38 | |
| *** coolsvap_ is now known as coolsvap | 07:39 | |
| prmtl | hi, is there anything else that need to be done to merge this request (new project): https://review.openstack.org/#/c/158263/? | 07:40 |
| *** sdake_ has quit IRC | 07:40 | |
| *** hdd has quit IRC | 07:40 | |
| *** FallenPegasus has joined #openstack-infra | 07:41 | |
| *** scheuran has joined #openstack-infra | 07:43 | |
| *** MarkAtwood has quit IRC | 07:44 | |
| *** FallenPegasus is now known as MarkAtwood | 07:44 | |
| *** armax has quit IRC | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:50 | |
| *** mrmartin has joined #openstack-infra | 07:50 | |
| *** vilobhmm has joined #openstack-infra | 07:51 | |
| *** sergsh has joined #openstack-infra | 07:51 | |
| *** HeOS has quit IRC | 07:53 | |
| AJaeger | prmtl, patience ;) | 07:53 |
| prmtl | AJaeger: ok :) | 07:53 |
| *** vilobhmm has quit IRC | 07:53 | |
| AJaeger | As every patch, it needs review by another core - you submitted yesterday and our turn around is not that fast, expect around a week. | 07:55 |
| *** Ala has joined #openstack-infra | 07:55 | |
| *** Ryan_Lane has quit IRC | 07:55 | |
| AJaeger | prmtl, if it's urgent, please ping another core - most are in the US timezone. | 07:56 |
| *** MarkAtwood has quit IRC | 07:56 | |
| prmtl | AJaeger: thanks for help | 07:56 |
| *** sputnik13 has quit IRC | 07:59 | |
| *** tnovacik_ has joined #openstack-infra | 07:59 | |
| *** woodster_ has quit IRC | 08:00 | |
| *** sputnik13 has joined #openstack-infra | 08:00 | |
| *** unicell has joined #openstack-infra | 08:01 | |
| *** jcoufal has joined #openstack-infra | 08:01 | |
| *** mpavone has joined #openstack-infra | 08:03 | |
| *** MaxV has joined #openstack-infra | 08:04 | |
| *** jcoufal_ has quit IRC | 08:05 | |
| *** abhishek has joined #openstack-infra | 08:09 | |
| *** luqas has joined #openstack-infra | 08:09 | |
| *** otter768 has joined #openstack-infra | 08:11 | |
| *** arxcruz has joined #openstack-infra | 08:13 | |
| *** MaxV has quit IRC | 08:13 | |
| *** mrmartin has quit IRC | 08:13 | |
| *** luqas has quit IRC | 08:14 | |
| *** hichihara has quit IRC | 08:15 | |
| *** otter768 has quit IRC | 08:16 | |
| BharatK | Hi as per clarkb suggestion | 08:26 |
| BharatK | deepakcs https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1803 is the shell the job runs after https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n326 | 08:26 |
| BharatK | But while running locally I am facing below error | 08:27 |
| *** ddieterly has joined #openstack-infra | 08:27 | |
| BharatK | Triggered by: https://review.openstack.org/ patchset | 08:27 |
| BharatK | Pipeline: | 08:27 |
| BharatK | Available disk space on this host: | 08:27 |
| BharatK | Filesystem Size Used Avail Use% Mounted on | 08:27 |
| BharatK | /dev/xvda1 158G 2.8G 149G 2% / | 08:27 |
| BharatK | devtmpfs 3.9G 0 3.9G 0% /dev | 08:27 |
| BharatK | tmpfs 3.7G 0 3.7G 0% /dev/shm | 08:27 |
| BharatK | tmpfs 3.7G 8.4M 3.7G 1% /run | 08:27 |
| BharatK | tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup | 08:27 |
| BharatK | Setting up the host | 08:27 |
| BharatK | ... this takes a few seconds (logs at logs/devstack-gate-setup-host.txt.gz) | 08:27 |
| BharatK | Setting up the workspace | 08:27 |
| BharatK | ... this takes 3 - 5 minutes (logs at logs/devstack-gate-setup-workspace-new.txt.gz) | 08:27 |
| BharatK | Running gate_hook | 08:27 |
| BharatK | Job timeout set to: 109 minutes | 08:27 |
| BharatK | 2015-02-24 07:57:05,141 ERROR: branch not allowed by features matrix: {branch-override} | 08:27 |
| BharatK | ERROR: the main setup script run by this job failed - exit code: 1 | 08:27 |
| BharatK | please look at the relevant log files to determine the root cause | 08:27 |
| BharatK | Cleaning up host | 08:28 |
| BharatK | ... this takes 3 - 4 minutes (logs at logs/devstack-gate-cleanup-host.txt.gz) | 08:28 |
| BharatK | Can any one please help to resolve this issue | 08:28 |
| deepakcs | BharatK, pls use http://pastebin.com/ | 08:28 |
| BharatK | Thanks deepakcs, http://paste.openstack.org/show/181062/ | 08:29 |
| deepakcs | AJaeger, any idea about the above error (ERROR: branch not allowed by features matrix: {branch-override}) while running ./safe-devstack-vm-gate-wrap.sh | 08:30 |
| AJaeger | deepakcs, you did not set the macro correctly, {branch-override} is a macro that needs to be expanded. | 08:31 |
| *** ddieterly has quit IRC | 08:31 | |
| deepakcs | AJaeger, ah ok, so any idea what it expands to. We are just trying to re-create the same tempest env that upstream infra VM has, in order to reproduce the oom issue, and we are blocked by this now | 08:32 |
| AJaeger | you're using '{pipeline}-tempest-dsvm-full-glusterfs-centos7{branch-designator}' ? | 08:33 |
| AJaeger | So, {branch-designator} should be "" | 08:34 |
| *** mpaolino has joined #openstack-infra | 08:34 | |
| AJaeger | and {branch-override} is default | 08:35 |
| deepakcs | AJaeger, thanks we will try with that. are these related to git branches or somethign else ? Sorry i don't fully understnad how they work in the CI env | 08:37 |
| *** dtantsur|afk is now known as dtantsur | 08:37 | |
| *** teran has quit IRC | 08:38 | |
| AJaeger | those are settings for devstack - and are related to how git is setup | 08:38 |
| AJaeger | sorry, I have some other stuff to do and can't be of much help right now. | 08:38 |
| deepakcs | AJaeger, ok, so "" and default are safe values for those vars, i assume | 08:39 |
| AJaeger | yep | 08:39 |
| deepakcs | AJaeger, np, thanks for ur time | 08:39 |
| *** oomichi_ has joined #openstack-infra | 08:40 | |
| *** jlibosva has joined #openstack-infra | 08:40 | |
| *** teran has joined #openstack-infra | 08:41 | |
| *** achanda has quit IRC | 08:42 | |
| *** skolekonov has joined #openstack-infra | 08:44 | |
| *** MaxV has joined #openstack-infra | 08:45 | |
| *** teran has quit IRC | 08:45 | |
| BharatK | AJaeger, test-matrix.py: error: argument -b/--branch: expected one argument | 08:45 |
| deepakcs | jhesketh, SergeyLukjanov ^^ can u help us in the above issue BharatK is seeing ? | 08:47 |
| *** sergsh has quit IRC | 08:48 | |
| *** achanda has joined #openstack-infra | 08:49 | |
| *** vigneshvar has joined #openstack-infra | 08:53 | |
| *** jp_at_hp has joined #openstack-infra | 08:54 | |
| *** jistr has joined #openstack-infra | 08:54 | |
| *** jp_at_hp1 has joined #openstack-infra | 08:59 | |
| *** dalgaaf has joined #openstack-infra | 08:59 | |
| *** jawed has joined #openstack-infra | 09:00 | |
| *** jp_at_hp has quit IRC | 09:00 | |
| *** sressot has quit IRC | 09:04 | |
| *** abhishek has quit IRC | 09:05 | |
| *** btran has quit IRC | 09:05 | |
| *** Murad has quit IRC | 09:05 | |
| *** omrim has quit IRC | 09:05 | |
| *** achanda has quit IRC | 09:06 | |
| *** armax has joined #openstack-infra | 09:06 | |
| *** HeOS has joined #openstack-infra | 09:06 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:15 | |
| *** sputnik13 has quit IRC | 09:15 | |
| *** dizquierdo has joined #openstack-infra | 09:16 | |
| *** markvoelker has joined #openstack-infra | 09:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Add fuel-tasks-validator project to stackforge https://review.openstack.org/158263 | 09:19 |
| *** sdake_ has joined #openstack-infra | 09:19 | |
| *** viktors|afk is now known as viktors | 09:20 | |
| *** derekh has joined #openstack-infra | 09:20 | |
| *** salv-orlando has joined #openstack-infra | 09:21 | |
| *** markus_z has joined #openstack-infra | 09:21 | |
| *** markvoelker has quit IRC | 09:22 | |
| *** sputnik13 has joined #openstack-infra | 09:22 | |
| *** tnovacik has quit IRC | 09:24 | |
| *** ddieterly has joined #openstack-infra | 09:28 | |
| *** yolanda has quit IRC | 09:29 | |
| *** teran has joined #openstack-infra | 09:29 | |
| *** yolanda has joined #openstack-infra | 09:29 | |
| *** doude has joined #openstack-infra | 09:31 | |
| *** ddieterly has quit IRC | 09:32 | |
| *** teran has quit IRC | 09:33 | |
| *** stevemar has quit IRC | 09:34 | |
| *** openstackgerrit has quit IRC | 09:36 | |
| *** openstackgerrit has joined #openstack-infra | 09:36 | |
| *** sergsh has joined #openstack-infra | 09:37 | |
| *** teran has joined #openstack-infra | 09:38 | |
| *** cody-somerville has joined #openstack-infra | 09:39 | |
| *** cody-somerville has quit IRC | 09:39 | |
| *** cody-somerville has joined #openstack-infra | 09:39 | |
| *** lucasagomes has joined #openstack-infra | 09:40 | |
| lucasagomes | hi all, if you have a time please take a look at https://review.openstack.org/#/c/158251 . It's needed for Ironic so we can start testing the a new default ramdisk in gate, it already has a +2 and the Ironic PTL also weighted in. Thank you | 09:43 |
| *** tnovacik has joined #openstack-infra | 09:45 | |
| *** tnovacik has quit IRC | 09:45 | |
| *** rlandy has joined #openstack-infra | 09:49 | |
| deepakcs | anteaya, can u help in scheduling a manual run of the glusterfs CI job on hpcloud provider ? Last time fungi did it, it didn't have the cgroup set correctly, so it actually ran with 32G of ram. We need to run in 8G and see if it re-creates the oom issue that we see on rax | 09:51 |
| deepakcs | anteaya, to rule out provider specific issues , if any | 09:52 |
| ianw | deepakcs: i can also bring up a hpcloud instance if you'd like | 09:53 |
| prmtl | can I be added as a initial core to https://review.openstack.org/#/admin/groups/624,members and https://review.openstack.org/#/admin/groups/625,members? | 09:56 |
| deepakcs | ianw, we are already using 1 instance from you :) | 09:56 |
| ianw | deepakcs: actually, no i can't. i guess my account can't start a standard.2xlarge vm | 09:57 |
| deepakcs | ianw, oh ok. I heard that sometimes provider specific issues are possible and from the local folks here at times changing the job's provider has helped, hence the request to anteaya | 09:58 |
| *** jcoufal has quit IRC | 09:59 | |
| *** jcoufal has joined #openstack-infra | 09:59 | |
| ianw | deepakcs: provider issues definitely possible ... was the job seen failing on both though? | 09:59 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: zuul-swift-upload support recursive globs https://review.openstack.org/158514 | 10:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Enable swift logs for python jobs https://review.openstack.org/156521 | 10:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Upload python test results to swift https://review.openstack.org/156522 | 10:00 |
| deepakcs | ianw, no, til now its all rax except 1 on hpcloud, but there bcos of cgroup issue, it ran with 30G ram, so i wanted to see what happens with 8G ram on hpcloud | 10:00 |
| *** _nadya_ has joined #openstack-infra | 10:00 | |
| ianw | deepakcs: right, yeah standard.2xlarge is the one with 8cpus, and as you say 32g ram | 10:01 |
| deepakcs | ianw, with 30G ram on hpcloud, oom did not happen | 10:01 |
| deepakcs | ianw, ideally per fungi they limit the ram to 8 with cgroups, but looks like that limiting was broken for centos7, fungi said he has sent a patch to fix that | 10:02 |
| deepakcs | i don't have the link to his patch tho' | 10:02 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Run python-jobs in devstack https://review.openstack.org/158615 | 10:02 |
| *** pelix1 has joined #openstack-infra | 10:03 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** harlowja_away has quit IRC | 10:04 | |
| *** pelix1 is now known as pelix | 10:04 | |
| *** ssam2 has joined #openstack-infra | 10:05 | |
| *** ihrachyshka has joined #openstack-infra | 10:06 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/infra-specs: Spec for nodepool image update retries https://review.openstack.org/155305 | 10:06 |
| *** erikmwilson is now known as Guest77452 | 10:09 | |
| *** erikmwilson has joined #openstack-infra | 10:09 | |
| *** erikmwil- has joined #openstack-infra | 10:09 | |
| *** otter768 has joined #openstack-infra | 10:12 | |
| *** erikmwilson has quit IRC | 10:14 | |
| *** erikmwil- is now known as erikmwilson | 10:14 | |
| *** psedlak_clone has joined #openstack-infra | 10:14 | |
| *** psedlak_clone has quit IRC | 10:16 | |
| *** _nadya_ has quit IRC | 10:16 | |
| *** otter768 has quit IRC | 10:17 | |
| *** markvoelker has joined #openstack-infra | 10:18 | |
| openstackgerrit | Merged openstack-infra/storyboard: Authorization codes validate on TTL. https://review.openstack.org/155084 | 10:20 |
| *** markvoelker has quit IRC | 10:23 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Run python-jobs in devstack https://review.openstack.org/158615 | 10:23 |
| *** sdake__ has joined #openstack-infra | 10:25 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: The email publisher needs a list of recipients https://review.openstack.org/153432 | 10:26 |
| *** armax has quit IRC | 10:26 | |
| *** sdake___ has joined #openstack-infra | 10:27 | |
| *** sdake_ has quit IRC | 10:28 | |
| *** marcusvrn has joined #openstack-infra | 10:28 | |
| *** ddieterly has joined #openstack-infra | 10:29 | |
| *** sdake__ has quit IRC | 10:30 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/python-storyboardclient: Added tags controller https://review.openstack.org/157857 | 10:30 |
| *** yfried__ is now known as yfried|afk | 10:31 | |
| *** armax has joined #openstack-infra | 10:32 | |
| *** marcusvrn1 has joined #openstack-infra | 10:32 | |
| *** ddieterly has quit IRC | 10:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add Fedora 21 image build to nodepool https://review.openstack.org/140911 | 10:33 |
| *** marcusvrn has quit IRC | 10:35 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Run Shade jobs in devstack https://review.openstack.org/158615 | 10:35 |
| *** tnovacik has joined #openstack-infra | 10:37 | |
| *** jamielennox is now known as jamielennox|away | 10:42 | |
| *** cody-somerville has quit IRC | 10:50 | |
| vponomaryov | sdague, fungi, AJaeger, anteya: according to devstack's ERROR_ON_CLONE and change https://review.openstack.org/#/c/158075/ here is latest attempt to use devstack plugin - http://logs.openstack.org/54/158054/6/check/gate-manila-tempest-dsvm-neutron/f5dbaa4/logs/devstacklog.txt.gz#_2015-02-24_10_50_02_697 So, plugin can not be fetched, even from gerrit | 10:55 |
| vponomaryov | sdague, fungi, AJaeger, anteya: I agree that direct clone from github is mistake, but plugin can not be fetched at all. | 10:56 |
| *** jawed has quit IRC | 10:56 | |
| vponomaryov | is there any project that already uses devstack plugin's in CI? | 10:57 |
| *** prmtl has left #openstack-infra | 10:57 | |
| openstackgerrit | Alan Pevec proposed openstack/requirements: Do not sync requirements to possibly capped library's master branch https://review.openstack.org/158086 | 10:58 |
| *** jamielennox|away is now known as jamielennox | 10:58 | |
| *** tnurlygayanov has joined #openstack-infra | 10:58 | |
| *** koolhead17 has quit IRC | 11:01 | |
| *** jamielennox is now known as jamielennox|away | 11:01 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/requirements: Block eventlet 0.17.0 https://review.openstack.org/158287 | 11:02 |
| *** jamielennox|away is now known as jamielennox | 11:02 | |
| marcusvrn1 | asselin: clarkb: Hi!! I'm trying to setup the local pypi repository, but when CI job runs, it's still connecting to pypi.org to download the packages. I edited the /home/jenkins/.pip/pip.conf file to redirect to my repository, but it didn't work | 11:02 |
| ianw | vponomaryov: glusterfs | 11:04 |
| vponomaryov | sdague, fungi, AJaeger, anteya: for example, devstack plugin in Ci for project "stackforge/ec2api", but they do not use gerrit and its refs, they use 'master' branch from "git.openstack.org". I am not sure it is corect | 11:05 |
| vponomaryov | ianw: can you provide a link, please? | 11:05 |
| *** ghostpl_ has joined #openstack-infra | 11:06 | |
| *** jamielennox is now known as jamielennox|away | 11:08 | |
| *** yfried|afk is now known as yfried__ | 11:08 | |
| marcusvrn1 | asselin: clarkb: and it was working a few days ago... | 11:08 |
| vponomaryov | sdague, fungi, AJaeger, anteya: for the moment, I guess the only difference between working devstack plugins and our not working is that directory for project is absent. So, other projects just do not do clone indeed. | 11:09 |
| vponomaryov | sdague, fungi, AJaeger, anteya: or correct would be to say - devstack itself do not do clone operation, because directory with project already exist and RECLONE=False is set | 11:10 |
| *** jamielennox|away is now known as jamielennox | 11:10 | |
| sdague | vponomaryov: the ec2api does use gerrit | 11:10 |
| vponomaryov | sdague: here is logs for ec2-api - http://logs.openstack.org/22/158622/1/check/check-tempest-dsvm-ec2api/3f3763a/logs/devstacklog.txt.gz | 11:10 |
| sdague | vponomaryov: for testing, correct, devstack doesn't do the cloning. zuul does the cloning before devstack gets to it | 11:10 |
| vponomaryov | where we have "git_clone git://git.openstack.org/stackforge/ec2-api /opt/stack/new/ec2-api master" | 11:10 |
| vponomaryov | so, we have cause detected, why zuul do not clone manila project? | 11:11 |
| openstackgerrit | Andrey Pavlov proposed openstack/requirements: Update tempest-lib version. https://review.openstack.org/158637 | 11:11 |
| vponomaryov | before devstack run? | 11:11 |
| sdague | yes, it's cloned before the devstack run | 11:11 |
| sdague | if you look at the git_clone function, you'll see that it *never* got to the actually cloning part | 11:11 |
| sdague | http://logs.openstack.org/22/158622/1/check/check-tempest-dsvm-ec2api/3f3763a/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-02-24_10_28_18_135 | 11:12 |
| sdague | that's where the cloning happens | 11:12 |
| sdague | vponomaryov: I also sent an email to the list describing this - http://lists.openstack.org/pipermail/openstack-dev/2015-February/056257.html | 11:12 |
| *** shardy is now known as shardy_afk | 11:13 | |
| vponomaryov | sdague: do you agree that,the fact clone operation was attempted to be performed mean directory is absent? | 11:14 |
| vponomaryov | project directory | 11:15 |
| *** ildikov has quit IRC | 11:15 | |
| sdague | yes, because you *did not* do the 2 things required in that email | 11:15 |
| *** koolhead17 has joined #openstack-infra | 11:15 | |
| sdague | you read that email right? | 11:16 |
| vponomaryov | yes | 11:16 |
| sdague | which explains what you need in your project definitions for this to work | 11:16 |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 11:16 |
| *** koolhead17 has quit IRC | 11:16 | |
| vponomaryov | sdague: 1 and 2 are done and third is done by direct update of localrc | 11:17 |
| sdague | vponomaryov: where? | 11:17 |
| vponomaryov | sdague, here is third - https://review.openstack.org/#/c/158054/6/contrib/ci/pre_test_hook.sh | 11:17 |
| sdague | vponomaryov: no, that's too later | 11:18 |
| *** salv-orlando has quit IRC | 11:18 | |
| vponomaryov | sdague: if so, how devstack even know about manila project? | 11:18 |
| sdague | also, I don't see anywhere that you change the PROJECTS list | 11:18 |
| vponomaryov | sdague: it is in devstack-gate repo | 11:18 |
| marcusvrn1 | asselin: clarkb: this is the error returned: http://paste.openstack.org/show/181152/ | 11:18 |
| *** markvoelker has joined #openstack-infra | 11:19 | |
| sdague | https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ec2-api.yaml#L29-L36 | 11:19 |
| sdague | vponomaryov: do that | 11:19 |
| vponomaryov | sdague: here is update of projects - https://github.com/openstack-infra/devstack-gate/blob/49a9167f/devstack-vm-gate-wrap.sh#L58 | 11:19 |
| vponomaryov | sdague, as you can see it is already done | 11:20 |
| *** dims has joined #openstack-infra | 11:20 | |
| ianw | vponomaryov: i think in https://review.openstack.org/#/c/158075/1/jenkins/jobs/manila.yaml you want to be setting PROJECTS ... before you run devsatck-vm-gate-wrap.sh | 11:20 |
| vponomaryov | ianw: why? before run of devsatck-vm-gate-wrap.sh we have only set up of env vars | 11:21 |
| sdague | vponomaryov: you messed up DEST somewhere | 11:22 |
| ianw | vponomaryov: see http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1803 | 11:22 |
| sdague | because by the time it tries to get to your clone point, it's failing because it's looking for manila in /opt/stack | 11:23 |
| sdague | and it should be looking in /opt/stack/new | 11:23 |
| lucasagomes | sdague, hi there, sorry to ping you directly. But when you get a time can you please take a look at https://review.openstack.org/#/c/158251 ? It's already +2'd and the Ironic PTL also weighted in. Would be good to get it in so we can go ahead with some tests in gate for the new ramdisk. Thanks | 11:24 |
| *** markvoelker has quit IRC | 11:24 | |
| vponomaryov | sdague, ianw: thanks for help, I will do same as here - https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ec2-api.yaml#L29-L36 | 11:25 |
| deepakcs | sdague, when u have some time, take a look at fungi 's observation for the gluster job oom issue and would appreciate your opinion on the same | 11:26 |
| sdague | vponomaryov: right, so you also need to remove the direct git_clone in your plugin | 11:27 |
| sdague | that's actually your issue right now | 11:27 |
| vponomaryov | sdague: it is already so in commit with its usage - https://review.openstack.org/#/c/158054/6/devstack/plugin.sh | 11:27 |
| deepakcs | sdague, i haven't found any strong data point for gluster being the cause for oom, but am open to other observations , so wanted to reuqest your eyes on the logs if that can help us get better data points | 11:28 |
| sdague | vponomaryov: you need to remove it for manila client as well | 11:28 |
| vponomaryov | sdague: NOTE - manilaclient is left there for cloning, because in local setup it will be absent | 11:28 |
| sdague | vponomaryov: no, you have to remove it, or it won't work | 11:28 |
| sdague | make an external plugin for manila client instead | 11:28 |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 11:28 |
| vponomaryov | sdague: since directory will exist, why should it fail? | 11:28 |
| ianw | vponomaryov: yeah, and sorry I didn't realise that the plugin was in the project, so I don't think you need the PROJECT reset, as you say it is in devstack-gate | 11:28 |
| sdague | because it won't exist | 11:28 |
| sdague | it will be in the *wrong* place | 11:29 |
| *** yamamoto has joined #openstack-infra | 11:29 | |
| *** ddieterly has joined #openstack-infra | 11:29 | |
| mescanef | hi, please add me (mmatuszkowiak@mirantis.com) to the fuel-plugin-mellanox group (https://review.openstack.org/#/c/157704/) | 11:31 |
| mescanef | thanks in advance | 11:31 |
| mescanef | :-) | 11:31 |
| *** cody-somerville has joined #openstack-infra | 11:31 | |
| vponomaryov | sdague: I still did not get, why extras.d hooks work when external devstack plugin does not, do you know? | 11:32 |
| *** skolekonov has quit IRC | 11:33 | |
| *** ddieterly has quit IRC | 11:34 | |
| sdague | vponomaryov: honestly, I don't know what's doing the chdir, but I saw it with the ec2 plugin, and we got around that. Honestly, it's probably best to propose something as close to their setup as possible. | 11:34 |
| *** mrmartin has joined #openstack-infra | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/python-storyboardclient: Added timeline controller https://review.openstack.org/158651 | 11:37 |
| *** jamielennox is now known as jamielennox|away | 11:38 | |
| *** jamielennox|away is now known as jamielennox | 11:41 | |
| *** mase_home has joined #openstack-infra | 11:41 | |
| *** GheRivero has quit IRC | 11:41 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/python-storyboardclient: Added user_tokens controller https://review.openstack.org/158652 | 11:41 |
| *** sputnik13 has quit IRC | 11:41 | |
| *** sputnik13 has joined #openstack-infra | 11:42 | |
| *** sputnik13 has quit IRC | 11:44 | |
| *** pc_m has joined #openstack-infra | 11:46 | |
| *** cdent has joined #openstack-infra | 11:48 | |
| *** dhritishikhar has joined #openstack-infra | 11:48 | |
| *** BobBall_AWOL is now known as BobBall | 11:48 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add support of external Devstack plugin for Manila https://review.openstack.org/158653 | 11:49 |
| vponomaryov | sdague ^ | 11:49 |
| openstackgerrit | Merged openstack-infra/project-config: Ironic: Bump memory size for non-concurrency job https://review.openstack.org/158251 | 11:49 |
| BobBall | What's the recommended way to look at stats generated by the zuul/jenkins combo? Getting the graphs that are on http://status.openstack.org/zuul/ would be a great start but I'm also interested in the high level questions like how many of these jobs ran, passed or failed | 11:50 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add support of external Devstack plugin for Manila https://review.openstack.org/158653 | 11:54 |
| *** spredzy is now known as spredzy|afk | 11:56 | |
| *** chlong has joined #openstack-infra | 11:56 | |
| *** mrmartin has quit IRC | 11:58 | |
| *** ildikov has joined #openstack-infra | 11:59 | |
| *** markvoelker has joined #openstack-infra | 12:05 | |
| *** rfolco has joined #openstack-infra | 12:07 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/python-storyboardclient: Added subscriptions controller https://review.openstack.org/158659 | 12:07 |
| *** salv-orlando has joined #openstack-infra | 12:10 | |
| *** markvoelker has quit IRC | 12:11 | |
| *** otter768 has joined #openstack-infra | 12:13 | |
| *** yamamoto has quit IRC | 12:15 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Added branches to storyboard https://review.openstack.org/150447 | 12:15 |
| *** koolhead17 has joined #openstack-infra | 12:17 | |
| *** otter768 has quit IRC | 12:18 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Added milestones https://review.openstack.org/152500 | 12:19 |
| Kiall | Q - Seeing apt mirror fails during some devstack runs, pointed at http://mirror.rackspace.com/ubuntu/, I thought OpenStack had their own mirrors? | 12:20 |
| *** oomichi_ has quit IRC | 12:21 | |
| *** yamamoto has joined #openstack-infra | 12:23 | |
| *** tnovacik has quit IRC | 12:29 | |
| *** aysyd has joined #openstack-infra | 12:29 | |
| *** yamamoto has quit IRC | 12:29 | |
| *** ddieterly has joined #openstack-infra | 12:30 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Fix openstack-specs ACLs typo https://review.openstack.org/158666 | 12:31 |
| *** ddieterly has quit IRC | 12:35 | |
| *** lucasagomes is now known as lucas-hungry | 12:36 | |
| *** zz_dimtruck is now known as dimtruck | 12:36 | |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Fix swift func tests to run using both auth versions https://review.openstack.org/158671 | 12:37 |
| *** _nadya_ has joined #openstack-infra | 12:43 | |
| *** adalbas has joined #openstack-infra | 12:44 | |
| *** chlong has quit IRC | 12:44 | |
| fungi | Kiall: no, we use the distro package mirrors run by our service providers. we have our own git mirrors and pypi mirrors but nobody has finished setting up distro package mirrors | 12:48 |
| Kiall | Ah, I thought OS had our own :) Fair enough! | 12:49 |
| fungi | BobBall: those graphs are generated via graphite.openstack.org which also has the data you're talking about | 12:49 |
| *** adalbas has quit IRC | 12:50 | |
| BobBall | oh wow... ok... That looks fun... *Investigates* | 12:50 |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** ZZelle has quit IRC | 12:51 | |
| *** ZZelle has joined #openstack-infra | 12:52 | |
| fungi | deepakcs: just to catch you up on the memory limit thing, turns out i forgot we stopped using cgroups enforcement for that in favor of a kernel boot parameter when we switched to single-use workers, and that was broken in hpcloud for systems we recently switched to creating with diskimage-builder. same end result though https://review.openstack.org/158413 | 12:55 |
| fungi | not fixed yet | 12:55 |
| *** armax has quit IRC | 12:56 | |
| *** adalbas has joined #openstack-infra | 12:57 | |
| BobBall | fungi: Do you / can you use those for monitoring - e.g. email notifications when particular metrics fall below a given level? | 12:58 |
| *** mwagner_ has quit IRC | 12:58 | |
| *** mwagner_lap has quit IRC | 12:59 | |
| BobBall | or do you tend to rely on irc notifications from affected devs being the only known thing faster than the speed of light | 12:59 |
| *** arosen1 has quit IRC | 12:59 | |
| *** jgjl has joined #openstack-infra | 13:00 | |
| *** yamamoto has quit IRC | 13:01 | |
| fungi | arosen: upgraded to what version of jenkins? we're running 1.565.3 so can't say for sure that our plugins work with anything newer than that | 13:03 |
| *** kgiusti has joined #openstack-infra | 13:03 | |
| fungi | oh, clarkb answered that | 13:03 |
| fungi | deepakcs: would you want access to a vm which already ran your job, or a fresh one you can try to run it on? | 13:04 |
| *** jedimike has joined #openstack-infra | 13:06 | |
| *** markvoelker has joined #openstack-infra | 13:06 | |
| *** dkliban_afk is now known as dkliban | 13:08 | |
| *** markvoelker has quit IRC | 13:11 | |
| *** Longgeek has joined #openstack-infra | 13:12 | |
| deepakcs | fungi, hey | 13:13 |
| * deepakcs reads up | 13:13 | |
| fungi | mescanef: i've added you as the initial member for https://review.openstack.org/#/admin/groups/618,members and https://review.openstack.org/#/admin/groups/619,members as requested | 13:13 |
| *** vigneshvar has quit IRC | 13:13 | |
| *** spredzy|afk is now known as spredzy | 13:14 | |
| fungi | BobBall: the data in graphite is purely informational (as is what we have in cacti, puppetboard, et cetera) and we have no automated notification mechanisms around any of it | 13:14 |
| BobBall | *nod* ok. | 13:15 |
| deepakcs | fungi, what do u mean by 'same end result' ? in fact in hpcloud gluster job ran fine :) | 13:15 |
| fungi | yes, our users/developers notify us far faster than any automated monitoring system ever would ;) | 13:15 |
| fungi | deepakcs: i mean that having the boot-time memory limiting parameter missing on those workers had the same end result as if we had been relying on cgroups to enforce the memory limit and somehow broke it | 13:16 |
| fungi | deepakcs: end result being we're missing the memory cap for those workers in hpcloud and have to fix it | 13:16 |
| deepakcs | fungi, ah ok, diff approach tho | 13:16 |
| *** doude has quit IRC | 13:17 | |
| *** Adri2000 has quit IRC | 13:17 | |
| fungi | so, yes, the copy of your job i ran in hpcloud ran with 30gb ram available to it, as you suspected | 13:17 |
| deepakcs | fungi, can we get 2 VMs one in rax and one in hpcloud, I wanted to see if the provider change has anything to do with the fail | 13:17 |
| *** katyafervent is now known as katyafervent_awa | 13:17 | |
| *** Adri2000 has joined #openstack-infra | 13:17 | |
| fungi | deepakcs: well, right now they're still running with wildly different amounts of available ram too | 13:17 |
| *** yfried__ is now known as yfried|afk | 13:17 | |
| *** doude has joined #openstack-infra | 13:17 | |
| deepakcs | fungi, so is it possible to use cgroups cmd to limit the ram before we start the job ? | 13:17 |
| *** dprince has joined #openstack-infra | 13:18 | |
| *** jamielennox is now known as jamielennox|away | 13:18 | |
| fungi | deepakcs: it's possible, or perhaps just modify the boot configuration and reboot the worker | 13:18 |
| fungi | which would likely be easier | 13:19 |
| deepakcs | fungi, will need steps if u want us (me and BharatK ) to do it. | 13:19 |
| fungi | deepakcs: i can do it once i get coffee and settle in to check e-mail. still waking up and doing morning routine stuff | 13:20 |
| *** jamielennox|away is now known as jamielennox | 13:20 | |
| *** yfried|afk is now known as yfried__ | 13:20 | |
| deepakcs | fungi, sure, no hurries, its evening here so myself and BharatK will ride back home | 13:20 |
| *** amotoki has quit IRC | 13:21 | |
| deepakcs | fungi, and then connect with you ? if u don't find us , can u send us email in the same thread ? | 13:21 |
| deepakcs | fungi, BharatK plans to work on the VMs u provide to see if it re-creates | 13:22 |
| deepakcs | fungi, but we definitely need 1 VM in rax, since we have't been able to see the failure ourselves yet! | 13:22 |
| fungi | deepakcs: BharatK: if you can point me at a public ssh key you want authorized i'll get you access to an unused worker vm in each provider (and fix the memory cap on the one in hpcloud) | 13:22 |
| *** mbacchi has joined #openstack-infra | 13:22 | |
| fungi | though we can't keep them around for more than 24 hours or so before we'll need to return them to the worker pool | 13:22 |
| deepakcs | fungi, ianw helped us get a rax VM with 8g ram using his personal accoutn (thanks ianw !) and we ran there twice, but it hasn't re-created for us :( | 13:23 |
| fungi | deepakcs: i'll also put together a quick paste with the exact commands i was running on these to recreate the problem | 13:24 |
| deepakcs | fungi, that would help a lot, u can infact send that along with the ssh info to the same email thread ? | 13:24 |
| *** armax has joined #openstack-infra | 13:24 | |
| fungi | sure | 13:25 |
| deepakcs | fungi, http://paste.fedoraproject.org/189560/75782714/ (for the pub keys of me and BharatK ) | 13:25 |
| *** radez_g0n3 is now known as radez | 13:25 | |
| *** salv-orlando has quit IRC | 13:25 | |
| *** salv-orlando has joined #openstack-infra | 13:26 | |
| *** adalbas has quit IRC | 13:26 | |
| deepakcs | fungi, ^^ in case u missed | 13:26 |
| *** bhunter71 has joined #openstack-infra | 13:26 | |
| fungi | deepakcs: thanks, i'll add those for you | 13:27 |
| deepakcs | fungi, thanks fungi. also if the ssh session becomes un-responsive, is there a way to get console access to these provider VMs ? | 13:28 |
| *** lucas-hungry is now known as lucasagomes | 13:28 | |
| *** gordc has joined #openstack-infra | 13:28 | |
| jroll | fungi: apparently there's still thawte certs in requests' bundle, though some were removed, and trying to match fingerprints is getting me nothing :/ | 13:30 |
| deepakcs | fungi, need to leave, send console info too, in case available, when u reply over the thread | 13:30 |
| *** deepakcs has quit IRC | 13:30 | |
| *** ddieterly has joined #openstack-infra | 13:31 | |
| *** BharatK has quit IRC | 13:31 | |
| *** ddieterly has quit IRC | 13:35 | |
| *** ddieterly has joined #openstack-infra | 13:35 | |
| *** skolekonov has joined #openstack-infra | 13:35 | |
| *** jaypipes has quit IRC | 13:36 | |
| *** skolekonov has quit IRC | 13:39 | |
| *** dhritishikhar has quit IRC | 13:39 | |
| *** skolekonov has joined #openstack-infra | 13:39 | |
| *** mase_home has quit IRC | 13:39 | |
| *** _nadya_ has quit IRC | 13:40 | |
| *** xyang1 has joined #openstack-infra | 13:40 | |
| *** woodster_ has joined #openstack-infra | 13:44 | |
| *** unicell1 has joined #openstack-infra | 13:44 | |
| *** ChuckC has joined #openstack-infra | 13:45 | |
| *** tnovacik_ has quit IRC | 13:46 | |
| *** unicell has quit IRC | 13:46 | |
| *** mwagner_lap has joined #openstack-infra | 13:47 | |
| *** mwagner_ has joined #openstack-infra | 13:47 | |
| mescanef | fungi: thanks | 13:48 |
| *** dkranz has joined #openstack-infra | 13:52 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** amitgandhinz has joined #openstack-infra | 13:55 | |
| *** amitgandhinz has quit IRC | 13:55 | |
| *** amitgandhinz has joined #openstack-infra | 13:55 | |
| *** emagana has joined #openstack-infra | 13:56 | |
| *** bknudson has joined #openstack-infra | 13:57 | |
| *** carl_baldwin has joined #openstack-infra | 13:57 | |
| *** radez is now known as radez_g0n3 | 13:59 | |
| *** sdake_ has joined #openstack-infra | 13:59 | |
| *** vigneshvar has joined #openstack-infra | 13:59 | |
| *** mwagner_ has quit IRC | 14:00 | |
| *** arxcruz has quit IRC | 14:03 | |
| *** sdake___ has quit IRC | 14:03 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Tweak neutron project review stats https://review.openstack.org/156859 | 14:04 |
| *** dimtruck is now known as zz_dimtruck | 14:04 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Add Marek Denis to keystone core https://review.openstack.org/156026 | 14:04 |
| openstackgerrit | Merged openstack-infra/reviewstats: Sync core reviewer list for Trove https://review.openstack.org/156907 | 14:06 |
| openstackgerrit | Merged openstack-infra/reviewstats: Update the subproject list for Trove https://review.openstack.org/156908 | 14:06 |
| *** amitgandhinz has quit IRC | 14:06 | |
| *** markvoelker has joined #openstack-infra | 14:07 | |
| *** amitgandhinz has joined #openstack-infra | 14:07 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Deactivate Robert Collins from TripleO https://review.openstack.org/156724 | 14:07 |
| *** mrmartin has joined #openstack-infra | 14:07 | |
| *** ddieterly has quit IRC | 14:07 | |
| *** kgiusti has left #openstack-infra | 14:07 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Add lyz to core reviews in infra.json https://review.openstack.org/155532 | 14:08 |
| *** ghostpl_ has quit IRC | 14:09 | |
| *** yfried has joined #openstack-infra | 14:10 | |
| *** yfried__ has quit IRC | 14:10 | |
| *** erikmwilson is now known as Guest86093 | 14:11 | |
| *** erikmwil_ has joined #openstack-infra | 14:11 | |
| *** Guest86093 has quit IRC | 14:11 | |
| *** erikmwil_ is now known as erikmwilson | 14:11 | |
| *** erikmwil- has joined #openstack-infra | 14:11 | |
| *** amitgandhinz has quit IRC | 14:11 | |
| *** sdake_ has quit IRC | 14:11 | |
| *** sdake_ has joined #openstack-infra | 14:12 | |
| *** dims has quit IRC | 14:13 | |
| *** baoli has joined #openstack-infra | 14:13 | |
| *** changbl has quit IRC | 14:13 | |
| *** dims has joined #openstack-infra | 14:13 | |
| *** _nadya_ has joined #openstack-infra | 14:14 | |
| *** otter768 has joined #openstack-infra | 14:14 | |
| *** erikmwilson has quit IRC | 14:16 | |
| *** erikmwil- is now known as erikmwilson | 14:16 | |
| *** bcourt has quit IRC | 14:18 | |
| *** matrohon has joined #openstack-infra | 14:19 | |
| openstackgerrit | Sergey Kulanov proposed openstack-infra/project-config: Create Influxdb-Grafana Fuel plugin project in Stackforge https://review.openstack.org/156245 | 14:19 |
| *** otter768 has quit IRC | 14:19 | |
| matrohon | hi all | 14:19 |
| *** dustins has joined #openstack-infra | 14:21 | |
| *** ghostpl_ has joined #openstack-infra | 14:21 | |
| matrohon | can someone add me as member of the group networking-bgpvpn-core and networking-bgpvpn-release in gerrit? | 14:21 |
| *** vigneshvar has quit IRC | 14:21 | |
| matrohon | the new stackforge project bgpvpn has just been created | 14:22 |
| matrohon | ^^ networking-bgpvpn | 14:22 |
| *** mfink_ has joined #openstack-infra | 14:22 | |
| *** doug-fish has joined #openstack-infra | 14:23 | |
| *** zul has quit IRC | 14:23 | |
| matrohon | anteaya : hi | 14:23 |
| fungi | jroll: found it. grab thawte ssl root 3 from https://www.thawte.com/roots/index.html | 14:23 |
| *** katyafervent_awa is now known as katyafervent | 14:23 | |
| anteaya | matrohon: hello | 14:23 |
| fungi | jroll: then do `openssl s_client -connect identity.api.rackspacecloud.com:https -CAfile thawte_Server_CA.pem` | 14:23 |
| anteaya | matrohon: I don't have gerrit admin permissions, so I can't add people to gerrit groups | 14:23 |
| fungi | jroll: it checks out | 14:23 |
| fungi | jroll: so i'm guessing that one got dropped from requests/mkcert | 14:24 |
| matrohon | anteaya : thanks for approving the project networking-bgpvpn in stackforge | 14:24 |
| *** amotoki has joined #openstack-infra | 14:24 | |
| matrohon | anteaya : ack | 14:24 |
| anteaya | when fungi has a chance to do so he will probably get to it | 14:24 |
| anteaya | matrohon: that's my job | 14:24 |
| *** salv-orlando has quit IRC | 14:24 | |
| *** zul has joined #openstack-infra | 14:24 | |
| *** ghostpl_ has quit IRC | 14:25 | |
| fungi | matrohon: i've added you as the initial member of https://review.openstack.org/#/admin/groups/620,members and https://review.openstack.org/#/admin/groups/621,members now | 14:27 |
| matrohon | fungi : thank you very much! | 14:28 |
| *** sdake__ has joined #openstack-infra | 14:28 | |
| *** koolhead17 has quit IRC | 14:28 | |
| jroll | fungi: hrm, I guess just add that to the requests issue? | 14:29 |
| fungi | jroll: hrm, no | 14:29 |
| jroll | fungi: it worries me more that the cert is no longer trusted by mozilla and friends | 14:30 |
| fungi | jroll: when i retrieve https://raw.githubusercontent.com/kennethreitz/requests/b86330d1f15f04e77093800efb8c6bd6909e8c8e/requests/cacert.pem i can similarly substitute it and it verifies via openssl | 14:30 |
| jroll | x.x | 14:30 |
| jroll | so it's something else, then | 14:30 |
| openstackgerrit | Jan Hruban proposed openstack-infra/jenkins-job-builder: Add 'Build Flow Test Aggregator' publisher https://review.openstack.org/158083 | 14:31 |
| fungi | jroll: yeah, still digging | 14:31 |
| jroll | fungi: sorry, I have to step away for a few minutes, brb | 14:31 |
| fungi | k | 14:31 |
| jroll | thank you for looking at this | 14:31 |
| *** Sukhdev has joined #openstack-infra | 14:31 | |
| *** sdake_ has quit IRC | 14:31 | |
| *** skolekonov has quit IRC | 14:31 | |
| *** yfried has quit IRC | 14:32 | |
| *** yfried has joined #openstack-infra | 14:32 | |
| *** markvoelker has quit IRC | 14:32 | |
| *** markvoelker has joined #openstack-infra | 14:33 | |
| *** adalbas has joined #openstack-infra | 14:33 | |
| *** mtanino has quit IRC | 14:34 | |
| *** zz_dimtruck is now known as dimtruck | 14:35 | |
| *** dhritishikhar has joined #openstack-infra | 14:35 | |
| *** kgiusti has joined #openstack-infra | 14:36 | |
| *** yamahata has joined #openstack-infra | 14:36 | |
| *** sdake_ has joined #openstack-infra | 14:36 | |
| *** yfried has quit IRC | 14:37 | |
| *** yfried_ has joined #openstack-infra | 14:37 | |
| *** ayoung has joined #openstack-infra | 14:37 | |
| *** markvoelker has quit IRC | 14:38 | |
| *** dmsimard_away is now known as dmsimard | 14:38 | |
| *** abhirc has joined #openstack-infra | 14:38 | |
| *** doude has quit IRC | 14:39 | |
| *** _nadya_ has quit IRC | 14:39 | |
| *** abhirc has quit IRC | 14:39 | |
| *** sdake__ has quit IRC | 14:40 | |
| jroll | fungi: I'm back if you need a hand at all | 14:40 |
| fungi | jroll: just to make sure i'm not crazy, i double-checked and can definitely use novaclient with requests 69d9594 but not with requests a5a314e and the only difference between them is the requests/cacert.pem file | 14:41 |
| jroll | fungi: this is super strange | 14:42 |
| dstufft | jroll: fungi try installing certifi | 14:42 |
| dstufft | Also see also: https://github.com/kennethreitz/requests/issues/2455 | 14:42 |
| fungi | dstufft: yep, likely the same problem as the issue jroll opened against requests last night | 14:43 |
| jroll | fungi: ya, this looks like Lukasa walking through the same stuff | 14:44 |
| *** skolekonov has joined #openstack-infra | 14:44 | |
| fungi | dstufft: looks like you tested basically the same things we did too | 14:44 |
| fungi | with the same results | 14:44 |
| *** wenlock has joined #openstack-infra | 14:44 | |
| *** ddieterly has joined #openstack-infra | 14:44 | |
| *** koolhead17 has joined #openstack-infra | 14:45 | |
| *** ghostpl_ has joined #openstack-infra | 14:46 | |
| fungi | okay, that being the case, there's no new info we have to add, and i think the other issue can just be linked to 2455 and let other people finish hashing out what's wrong with the cert bundle there | 14:46 |
| fungi | (pun intended) | 14:47 |
| jroll | hehe | 14:47 |
| * fungi gets back to solving unrelated problems | 14:47 | |
| jroll | I agree | 14:47 |
| jroll | thanks for letting me derail you :) | 14:47 |
| *** mriedem has joined #openstack-infra | 14:47 | |
| fungi | no thanks needed. i'm just glad it's confirmed beyond merely me | 14:47 |
| *** tiswanso has joined #openstack-infra | 14:48 | |
| openstackgerrit | Jan Hruban proposed openstack-infra/jenkins-job-builder: Support additional features in Build Flow project https://review.openstack.org/158084 | 14:49 |
| *** erikmwilson has quit IRC | 14:49 | |
| *** erikmwilson has joined #openstack-infra | 14:49 | |
| *** erikmwil- has joined #openstack-infra | 14:50 | |
| anteaya | fungi: can you take a look at a freenode account entilted openstack-meetin | 14:51 |
| anteaya | it spoke up once and is popping in and out of channel | 14:52 |
| anteaya | not sure if it is innoucuous or what | 14:52 |
| anteaya | I'm in -meeting myself | 14:52 |
| fungi | seems to have left the channel | 14:54 |
| *** sdake__ has joined #openstack-infra | 14:55 | |
| anteaya | thanks for looking | 14:55 |
| fungi | looks like the connection was originating in a punjabi phone carrier's network | 14:56 |
| fungi | chances are someone was trying to join the meeting and typed the channel name into the nick field of their client or something | 14:57 |
| *** dimtruck is now known as zz_dimtruck | 14:57 | |
| fungi | the same ip address re-joined the channel a few minutes later with a nick of amsaha | 14:58 |
| openstackgerrit | Syed Ismail Faizan Barmawer proposed openstack-infra/project-config: Add ironic-lib to openstack https://review.openstack.org/157636 | 14:58 |
| anteaya | fungi: ah thanks | 14:58 |
| anteaya | that makes sense then | 14:59 |
| *** sdake_ has quit IRC | 14:59 | |
| *** tnovacik_ has joined #openstack-infra | 14:59 | |
| fungi | hanlon's razor | 14:59 |
| anteaya | seems so | 15:00 |
| *** nelsnelson has joined #openstack-infra | 15:01 | |
| dprince | morning infra. Might I get a second set of eyes on this patch to increase Puppet TripleO test coverage? https://review.openstack.org/#/c/157503/ | 15:02 |
| anteaya | dprince: I was just reading about you | 15:02 |
| anteaya | in the tripleO update email | 15:02 |
| *** sdake_ has joined #openstack-infra | 15:02 | |
| dprince | anteaya: oh, hopefully nothing too bad ;) | 15:03 |
| anteaya | I already reviewed that patch, so nothing more I can do here | 15:03 |
| dprince | anteaya: thanks, appreciated | 15:03 |
| anteaya | dprince: just noting how busy you are | 15:03 |
| anteaya | go you | 15:03 |
| dprince | anteaya: got to see lots of infra folk at the meetup. always good to see how things are going and catchup on the latest | 15:04 |
| dstufft | fungi: jroll are you using python 2 or 3 | 15:04 |
| *** emagana has quit IRC | 15:04 | |
| anteaya | dprince: nice! | 15:04 |
| fungi | dstufft: i was reproducing it with python 2.7 | 15:04 |
| *** yfried__ has joined #openstack-infra | 15:04 | |
| jroll | dstufft: 2.7 here as well | 15:04 |
| anteaya | dprince: yes hard to keep up with the latest, isn't it | 15:05 |
| jroll | dstufft: 2.7.8 specifically | 15:05 |
| *** emagana has joined #openstack-infra | 15:05 | |
| dstufft | fungi: 2.7.x? | 15:05 |
| fungi | dstufft: 2.7.6 here | 15:06 |
| dstufft | I can't repro with 2.7.9 (but i can with 3.4.2), let me try to pull down 2.7.8 and see what happens | 15:06 |
| *** sdake__ has quit IRC | 15:06 | |
| fungi | perhaps requests defers to python's internal cert validation with 2.7.9? | 15:07 |
| *** adalbas has quit IRC | 15:07 | |
| *** yfried_ has quit IRC | 15:07 | |
| *** alexpilotti has quit IRC | 15:07 | |
| dstufft | I think they disable it | 15:07 |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Added branches to storyboard https://review.openstack.org/150447 | 15:08 |
| *** sarob has joined #openstack-infra | 15:08 | |
| *** yfried has joined #openstack-infra | 15:08 | |
| *** dkranz has quit IRC | 15:10 | |
| *** emagana has quit IRC | 15:10 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Added milestones https://review.openstack.org/152500 | 15:10 |
| *** signed8bit has joined #openstack-infra | 15:10 | |
| *** yfried__ has quit IRC | 15:11 | |
| *** katyafervent is now known as katyafervent_awa | 15:11 | |
| openstackgerrit | Merged openstack/requirements: Do not sync requirements to possibly capped library's master branch https://review.openstack.org/158086 | 15:12 |
| BobBall | Heh - seems like we're all hitting the same issue. Just got a notification of your comment on the requests issue fungi! | 15:13 |
| BobBall | Found through a completely different route of course ;) | 15:14 |
| *** dangers_away is now known as dangers | 15:15 | |
| *** koolhead17 has quit IRC | 15:15 | |
| *** katyafervent_awa is now known as katyafervent | 15:15 | |
| BobBall | Downgrading to requests 2.5.1 fixes it, as does installing certifi which seems to still be working ATM | 15:16 |
| dstufft | yea, certifi works because it's using an older CA Bundle | 15:16 |
| dstufft | same with requests 2.5.1 | 15:16 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:16 | |
| BobBall | Ah - I now realise that it's you on the emails dstufft... I'd missed your surname! | 15:17 |
| *** salv-orlando has joined #openstack-infra | 15:17 | |
| dstufft | BobBall: :) | 15:17 |
| openstackgerrit | Nikolay Fedotov proposed openstack-infra/project-config: Functional test job for Kolla https://review.openstack.org/158726 | 15:18 |
| *** GheRivero has joined #openstack-infra | 15:18 | |
| BobBall | Thanks for the fixes :) | 15:18 |
| *** prad has joined #openstack-infra | 15:19 | |
| *** zz_dimtruck is now known as dimtruck | 15:19 | |
| *** imcsk8 has quit IRC | 15:19 | |
| *** imcsk8 has joined #openstack-infra | 15:20 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remove unnecessary warning message. https://review.openstack.org/158543 | 15:20 |
| *** arxcruz has joined #openstack-infra | 15:22 | |
| openstackgerrit | Nikolay Fedotov proposed openstack-infra/project-config: Functional test job for Kolla https://review.openstack.org/158726 | 15:23 |
| *** yfried_ has joined #openstack-infra | 15:23 | |
| *** mtanino has joined #openstack-infra | 15:23 | |
| *** devlaps has joined #openstack-infra | 15:24 | |
| *** devlaps has quit IRC | 15:24 | |
| *** dkranz has joined #openstack-infra | 15:25 | |
| *** ddieterl_ has joined #openstack-infra | 15:25 | |
| *** ddieterly has quit IRC | 15:25 | |
| *** deepakcs has joined #openstack-infra | 15:25 | |
| deepakcs | fungi, hi, any luck on getting the VMs ? | 15:26 |
| *** yfried__ has joined #openstack-infra | 15:26 | |
| *** yfried has quit IRC | 15:26 | |
| *** yfried_ has quit IRC | 15:28 | |
| fungi | deepakcs: yep, got a lot going on. i have them held for you but still need to adjust the memory on the hpcloud instance | 15:28 |
| *** adalbas has joined #openstack-infra | 15:28 | |
| deepakcs | fungi, ok, we will wait for ur mail then, i hope this will be available for us for 1-2 days, so that we can try our fullest to re-create ? | 15:29 |
| fungi | oh, i take that back. i ended up with two rackspace instances because i wasn't paying attention. i'll recycle one of these and grab one from hpcloud | 15:29 |
| deepakcs | fungi, sure :) | 15:29 |
| fungi | deepakcs: yeah, hopefully. sorry, trying to juggle too many requests at once | 15:29 |
| *** bcourt has joined #openstack-infra | 15:30 | |
| deepakcs | fungi, i understand, apprecaite your juggling :) | 15:30 |
| deepakcs | fungi, as discussed, send mail with the steps too, so that myself or bharat can takeover from there | 15:30 |
| fungi | yep, planning to do so | 15:30 |
| clarkb | were you able to reproduce locally using the scripts jenkins runs? | 15:32 |
| *** _nadya_ has joined #openstack-infra | 15:35 | |
| *** koolhead17 has joined #openstack-infra | 15:35 | |
| *** dprince has quit IRC | 15:36 | |
| *** jcoufal has quit IRC | 15:37 | |
| *** jcoufal has joined #openstack-infra | 15:37 | |
| fungi | sounds like not. ianw provided them with some instances from hpcloud | 15:37 |
| fungi | also we don't (yet) have confirmation that it fails in hpcloud, just observed in rackspace so far | 15:38 |
| deepakcs | clarkb, no, we tried twice and it didn't re-create | 15:39 |
| deepakcs | fungi, ianw provided us in rax cloud, using his personal account | 15:39 |
| clarkb | right but we tried doing what jenkins does? because last night the implication was we hadnt | 15:40 |
| deepakcs | clarkb, BTW the shell script u asked us to directly run, worked with some hacks (didn't work out of the box) | 15:40 |
| clarkb | if thats changed since then awesome | 15:40 |
| clarkb | you likely had to substitute some of the jjb vars manually | 15:41 |
| deepakcs | clarkb, yeah , also some issues with stack user and sudoeer perm stuff, finally we managed to get it working, it ran with few testcase failues, but didn't oom | 15:41 |
| deepakcs | clarkb, fungi till now we haven't been able to see the oom failure ourself (either locally or on the rax VM provided by ianw). | 15:42 |
| clarkb | how long did it take on the rax instance? about an hour? | 15:42 |
| deepakcs | clarkb, fungi My hope is to use the exact same steps that fungi used and on the same provider + hpcloud (just to rule out provider specicif issues, if any) | 15:42 |
| *** stevemar has joined #openstack-infra | 15:43 | |
| fungi | deepakcs: ianw: was it a 8gb performance hvpvm instance running ubuntu 14.04 lts? | 15:43 |
| deepakcs | clarkb, yes a little more than a hour for the first run. 2nd run took less time, but we observed lot of api.compute testcase failures, which weren't seen in first run | 15:43 |
| clarkb | no centos7 is the image type that should be used | 15:43 |
| clarkb | iirc | 15:43 |
| fungi | er, pvhvm | 15:43 |
| deepakcs | fungi, i remember ianw mentioning 'performance' vm but it was havign centos7 | 15:43 |
| fungi | oh,k right, centos7 | 15:43 |
| fungi | yeah | 15:44 |
| deepakcs | fungi, we are and must use centos7 | 15:44 |
| fungi | 8gb performance flavor with "CentOS 7 (PVHVM)" is what we're using in rackspace | 15:44 |
| deepakcs | clarkb, fungi one data point we do have is when fungi tried to run in hpcloud VM with 30G ram, and it ran fine, so with higher mem it doesn't oom | 15:44 |
| jroll | is there a straightforward way to find out which openstack projects use a given library, besides grep */requirements.txt? | 15:44 |
| *** yamamoto has quit IRC | 15:45 | |
| deepakcs | clarkb, fungi and looking at glusterfs logs we haven't figured anything that fatal to cause mem leak/hogs | 15:45 |
| clarkb | jroll not really, another option is to install each to a venv and pip freeze it | 15:45 |
| deepakcs | clarkb, fungi so currently, I am still clueless as to why we get into oom in 8g ram system. | 15:45 |
| jroll | clarkb: yeah, ok. thanks | 15:45 |
| *** erikmwilson has quit IRC | 15:45 | |
| *** erikmwil- is now known as erikmwilson | 15:45 | |
| pleia2 | good morning | 15:46 |
| fungi | deepakcs: well, that and the job runs for about 3 hours as long as we up the devstack-gate timeout sufficiently, so the system is straining in general | 15:46 |
| jeblair | fungi, clarkb, mtreinish: anyone looked into: Subject: ** WARNING: Filesystem on subunit2sql-MySQL ** | 15:46 |
| deepakcs | fungi, yes thats a fact, but then mysqld and httpd are the max mem consuming processes | 15:46 |
| fungi | jeblair: i have not noticed that yet, no | 15:47 |
| mtreinish | jeblair: I was not aware there was an issue | 15:47 |
| jeblair | fungi: if i'm reading that right, it seems to suggest we have undersized the trove mysql instance for subunit2sql and it is filling up; has 10% free | 15:47 |
| *** david-lyle_afk is now known as david-lyle | 15:47 | |
| fungi | deepakcs: however, keep in mind that some things (for example horizon and keystone) are running within the context of httpd too | 15:47 |
| deepakcs | fungi, and from IO perspective, i see swift and cinder as the major IO services + mysqld too, since it would use some disk for its DB | 15:47 |
| fungi | jeblair: that sounds likely. we guessed because we had no idea what the actual total would end up being and i think no expiration was planned into it yet | 15:48 |
| deepakcs | fungi, one thing consistently i see is mysqld getting killed, httpd sometimes and sometimes its nova-compute | 15:48 |
| jeblair | mtreinish, fungi: should we expire or grow? | 15:48 |
| mtreinish | jeblair: hmm, yeah I think we were always expecting to blow past it at some point | 15:48 |
| mtreinish | jeblair: I was really hoping to at least keep a releases worth of data in the db if possible | 15:49 |
| *** mrunge is now known as mrunge_afk | 15:49 | |
| fungi | i'm fine with either, whatever mtreinish things would be useful retention within reason | 15:49 |
| fungi | er, thinks | 15:49 |
| deepakcs | fungi, and all IO causing stuff uses loop device as their backends, so i 'm jsut wondering if we are stressing the block/LVM layer of centos7 so much that its oom'ing ? | 15:49 |
| *** mrunge_afk has quit IRC | 15:49 | |
| jeblair | mtreinish: by what factor should we multiply the current capacity? | 15:49 |
| *** markvoelker has joined #openstack-infra | 15:49 | |
| clarkb | jeblair at least 2 if we want one release per db | 15:50 |
| fungi | deepakcs: that's certainly possible. we're doing nested virtualization and our service providers don't provide much in the way of iops guarantees | 15:50 |
| *** zul has quit IRC | 15:50 | |
| mescanef | hi, please add me (mmatuszkowiak@mirantis.com) to the fuel-plugin-elasticsearch-kibana group (https://review.openstack.org/#/c/154422/) | 15:50 |
| clarkb | db was made in paris we are ~halfish way through cycle | 15:50 |
| *** zul has joined #openstack-infra | 15:50 | |
| mescanef | fungi: :-) | 15:50 |
| jeblair | clarkb: maybe 3 then, so we don't end up at 90% at end of cycle? | 15:50 |
| deepakcs | fungi, clarkb looking at dstat, syslog and the different fields of dstat , i see tons of object-* processes (of swift) on cpu most of the time, compared to gluster or cinder | 15:50 |
| clarkb | jeblair +1 | 15:50 |
| mtreinish | jeblair: sounds good | 15:50 |
| *** carl_baldwin has quit IRC | 15:50 | |
| fungi | deepakcs: glance is likely storing vm images in swift | 15:51 |
| fungi | deepakcs: but i don't honestly have a deep knowledge of devstack internals | 15:51 |
| mtreinish | I probably should come up with a sane expiration story too | 15:51 |
| deepakcs | fungi, possible, i wasn't sure whats the default backend of glance, but yes it will store in disk somewhere but glance is mostly limited to times when we boot for first time | 15:51 |
| *** carl_baldwin has joined #openstack-infra | 15:51 | |
| deepakcs | fungi, so i don't attribute glance to the IO load much | 15:52 |
| jeblair | mtreinish, clarkb, fungi: current size is 100GB, max size is, coincidentally, 300GB. resizing to 300. | 15:52 |
| deepakcs | fungi, I am going back to what sdague had first said.. maybe the whole combination of gluster, swift, mysqld, glance, httpd is causing an overkill on the 8g ram VM, thats its oom'ing | 15:53 |
| deepakcs | fungi, so tryign with something bigger than 8g ram could be one option, but before that i wanted to re-create the issue in front of me, then try increasing mem to prove one way or the other | 15:53 |
| jeblair | clarkb, fungi, mordred: also, wow? 300 is max? that seems un-large. | 15:53 |
| jeblair | mtreinish, clarkb, fungi: resize complete | 15:54 |
| fungi | jeblair: that sounds... serendipitous (hopefully we don't need more than a 3x increase!) | 15:54 |
| mordred | jeblair: I'm less and less convinced of the value of that service | 15:54 |
| *** sarob has quit IRC | 15:54 | |
| *** abhirc has joined #openstack-infra | 15:54 | |
| deepakcs | fungi, is it possible to get somethign bigger than 8g ram ? if yes, you can try to run there just to compare if say 10g or 16g helps gets past the issue, since you re-created it thrice :) | 15:55 |
| fungi | mordred: well, once we figured out that we should add a custom configuration so that it doesn't time out connections after 2 minutes (as compared to the mysql default of 8 hours) it's been surprisingly more stable at least | 15:55 |
| *** mattfarina has joined #openstack-infra | 15:55 | |
| *** BharatK has joined #openstack-infra | 15:56 | |
| *** carl_baldwin has quit IRC | 15:56 | |
| fungi | deepakcs: anything's possible, it's just a question of what we have time for and are willing to support. my goal is to help you reproduce the failure so you can find a way to get your job hopefully fitting within the constraints of the systems we already support | 15:56 |
| deepakcs | fungi, agreed | 15:56 |
| *** vilobhmm has joined #openstack-infra | 15:57 | |
| deepakcs | fungi, and hence i had the earlier tries where i skipped the backup tests, but it re-created even with that, which was a shock for me | 15:57 |
| deepakcs | fungi, again, recreated by you, didn't re-creates for us in the local VM | 15:57 |
| *** annegent_ has joined #openstack-infra | 15:58 | |
| mtreinish | fungi: yeah, I hope we won't either. If it comes down to it, we can just manually deploy a db I guess | 15:58 |
| deepakcs | fungi, r u aware of any other-than-default settings in the centos7 OS, something un-usual, given the status quo, i am now doubting everything thats there :) | 15:58 |
| *** abhirc_ has joined #openstack-infra | 15:58 | |
| fungi | deepakcs: nothing i'm aware of, but as i said before we have no actual voting jobs using those servers yet and haven't been supporting centos7 officially as a project | 15:59 |
| *** abhirc has quit IRC | 15:59 | |
| deepakcs | fungi, Hmm ok. Then i think i wil give a try once with ur VMs and if it doesn't re-create, I believe its worth to try on F20, what say ? | 16:00 |
| *** ddieterl_ has quit IRC | 16:01 | |
| *** ddieterly has joined #openstack-infra | 16:02 | |
| clarkb | so I think an important bit is reproducing the devstack environment which the devstack gate stuff from the job should've done | 16:02 |
| fungi | deepakcs: perhaps (or f21 now? i think we deleted f20 recently) but not sure the support situation for us is any better there | 16:02 |
| *** Sukhdev has quit IRC | 16:02 | |
| *** coolsvap is now known as coolsvap_ | 16:02 | |
| deepakcs | clarkb, we do believe we did re-create the same devstack env (given that we were seeing the same messages that the failed job's console.log shows) | 16:03 |
| deepakcs | clarkb, in fact in the 2nd run, we even set the PYTHONHASHSEED to one of the failing job's seeds | 16:03 |
| *** erikmwilson has quit IRC | 16:04 | |
| *** erikmwilson has joined #openstack-infra | 16:04 | |
| *** nfedotov has joined #openstack-infra | 16:04 | |
| clarkb | deepakcs: the hashseed shouldn't matter, its random for each tox run | 16:04 |
| *** erikmwil- has joined #openstack-infra | 16:04 | |
| *** sarob has joined #openstack-infra | 16:04 | |
| clarkb | deepakcs: I am more concerned about the set of services, number of services, and so on | 16:04 |
| clarkb | part of that is determined by number of CPUs and some of it is determined by the devstack config | 16:04 |
| deepakcs | clarkb, since we use concurrency=4, and we didn't re-create on 1st run, we thought of using the same seed to hopefully re-create the same test/load pattern | 16:04 |
| *** koolhead17 has quit IRC | 16:05 | |
| deepakcs | clarkb, The rax VM ianw gave had centos7, 8g ram, 8 vcps, so similar to the infra VM | 16:05 |
| clarkb | deepakcs: right and was that node where ou ran the devstack-gate script machiner? | 16:05 |
| deepakcs | clarkb, yeah, with some minor changes to make it work | 16:06 |
| *** AJaeger_away has joined #openstack-infra | 16:06 | |
| *** AJaeger_away has joined #openstack-infra | 16:06 | |
| *** skolekonov has quit IRC | 16:06 | |
| *** vilobhmm has quit IRC | 16:06 | |
| clarkb | fungi: for the grub finalize.d change, I just realized, we probably need to rebuild the grub config there manually | 16:06 |
| clarkb | fungi: unless the script does it already but that doesn't show up in the diff so I need to go read | 16:07 |
| clarkb | fungi: ya doesn't look like it does that, but the grub two switch should run the update grub config command (I really wish that dib would separate grub install from grub config update) | 16:07 |
| deepakcs | fungi, so will wait for ur mail with the details then, thanks for helping. clarkb thanks to you as well for all the support | 16:08 |
| *** carl_baldwin has joined #openstack-infra | 16:09 | |
| *** reed has joined #openstack-infra | 16:09 | |
| *** btran has joined #openstack-infra | 16:10 | |
| openstackgerrit | Merged openstack/requirements: Update version of neutronclient https://review.openstack.org/142657 | 16:11 |
| *** emagana has joined #openstack-infra | 16:12 | |
| jlvillal | Anyone here who can help fix what appears to be a broken check for the openstack/python-ironicclient ? | 16:12 |
| *** tmcpeak has joined #openstack-infra | 16:13 | |
| jlvillal | The gate-python-ironicclient-python34 always seems to time out. | 16:13 |
| *** dprince has joined #openstack-infra | 16:13 | |
| clarkb | jlvillal: have an example job log? | 16:13 |
| jlvillal | clarkb: http://logs.openstack.org/29/157929/2/check/gate-python-ironicclient-python34/0e889b6/ | 16:14 |
| jlvillal | and thanks for looking :) | 16:14 |
| fungi | deepakcs: i'll have the e-mail out shortly, but here are the reproduction steps with some comments about how i arrived at them http://paste.openstack.org/show/181295/ | 16:15 |
| fungi | BharatK: ^ | 16:15 |
| *** otter768 has joined #openstack-infra | 16:15 | |
| BharatK | Sure fungi thank you. | 16:15 |
| *** jamielennox is now known as jamielennox|away | 16:15 | |
| clarkb | jlvillal: looks like all of the time is spent running tox -epy27, should be able to reproduce that locally on a trusty ubuntu machine | 16:15 |
| *** mpaolino has quit IRC | 16:15 | |
| *** jamielennox|away is now known as jamielennox | 16:16 | |
| clarkb | jlvillal: if I had to guess, its something in ironicclient that isn't quite py34 safe and causes it to hang during testing | 16:16 |
| *** changbl has joined #openstack-infra | 16:16 | |
| deepakcs | fungi, export DEVSTACK_LOCAL_CONFIG="enable_plugin devstack-plugin-sheepdog git://git.openstack.org/stackforge/devstack-plugin-sheepdog" | 16:16 |
| jlvillal | clarkb: Is this a new test added? | 16:16 |
| deepakcs | fungi, ^^ thats wrong, were u running with that ? | 16:16 |
| jlvillal | clarkb: I guess I would assume it used to work. | 16:16 |
| clarkb | jlvillal: I don't know, would have to check git logs | 16:16 |
| deepakcs | fungi, it should be glusterfs plugin instead of sheepdog | 16:17 |
| fungi | deepakcs: oh, whoops, cut and pasted from the wrong lines. new paste, hold on | 16:17 |
| deepakcs | fungi, k :) | 16:17 |
| fungi | deepakcs: but no i was running the glusterfs ones | 16:17 |
| jlvillal | clarkb: I'm confused why you said 'tox -epy27'. Where do you see that? I see 'tox -v -epy34' | 16:17 |
| deepakcs | fungi, yeah, i realised u shud have been, bcos i could see the cinder with gluster info msgs | 16:17 |
| clarkb | jlvillal: ya that command | 16:17 |
| *** skolekonov has joined #openstack-infra | 16:18 | |
| *** rhe00 has quit IRC | 16:18 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Use sphinx extlinks to shorten links to Jenkins wiki https://review.openstack.org/158756 | 16:18 |
| fungi | deepakcs: BharatK: sorry, http://paste.openstack.org/show/181303/ is what i was running there | 16:19 |
| jlvillal | clarkb: Thanks for the info. I will try locally. | 16:19 |
| *** dhritishikhar has quit IRC | 16:19 | |
| deepakcs | fungi, cool , looks good | 16:19 |
| deepakcs | fungi, waiting for the rax and hpcloud vm IPs | 16:20 |
| *** otter768 has quit IRC | 16:20 | |
| deepakcs | fungi, u have our pub keys, right ? | 16:20 |
| fungi | BharatK: deepakcs: yep, i've already added you to jenkins@104.239.136.99 but am waiting for an hpcloud vm now (so don't know what ip address it'll have yet) | 16:20 |
| BharatK | fungi, small doubt, in which user I should run this script? | 16:21 |
| fungi | BharatK: jenkins | 16:21 |
| BharatK | fungi, okay | 16:21 |
| *** sabeen1 has joined #openstack-infra | 16:21 | |
| deepakcs | fungi, so jenkins user already exist on the machine ? | 16:21 |
| *** vigneshvar has joined #openstack-infra | 16:22 | |
| deepakcs | fungi, i think we were running as stack the last time we ran it | 16:22 |
| fungi | BharatK: deepakcs: yes, it exists and has the necessary permissions to run devstack-gate | 16:22 |
| deepakcs | fungi, oh ok, just saw that we are logging in as jenkins :) | 16:22 |
| fungi | stack user may not be able to do the same things by default | 16:22 |
| deepakcs | fungi, in ianw 's VM were were given root ssh, and we created the stack user and ran usign that user | 16:23 |
| *** dannywilson has joined #openstack-infra | 16:23 | |
| fungi | there are some specific sudoers settings on those machines which differ between the jenkins, stack and tempest users | 16:23 |
| deepakcs | fungi, no wonder we got across some sudoer's issues, we fixed them manually and proceeded | 16:23 |
| deepakcs | BharatK, can u start on the above VM please ? | 16:23 |
| BharatK | deepakcs, yeah I logged and starting... | 16:24 |
| deepakcs | BharatK, it would be good to see if we are able to start on the tempest runs, otherwise we can get fungi 's help. | 16:24 |
| deepakcs | BharatK, awesome, thanks | 16:24 |
| asselin | marcusvrn1, sounds like you hit the same issue we hit yesterday | 16:24 |
| fungi | deepakcs: BharatK: yeah, in that paste i linked a comment to the http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/prepare_node_devstack.sh script we use to preconfigure these machines, in case you need to try to more accurately recreate on another machine elsewhere (the machines i'm holding for you don't need it run because it's how they were set up) | 16:24 |
| deepakcs | fungi, awesome, thats good to know | 16:25 |
| deepakcs | fungi, so i can use that script to setup my local VM and then run ur fpaste'd script to get the accurate reproduction ? | 16:25 |
| fungi | deepakcs: yes, that's what i would recommend | 16:26 |
| marcusvrn1 | asselin: I think it's intermittent, because now it's working well.. | 16:26 |
| deepakcs | fungi, nice, i think this is very good info for us. | 16:26 |
| fungi | hpcloud is taking a while to boot these. sitting in a building state in nodepool for 20 minutes and counting | 16:26 |
| asselin | marcusvrn1, ok...i've made a bit of progress, but not 100% back yet. | 16:27 |
| marcusvrn1 | asselin: did you do something to try to fix it up | 16:27 |
| marcusvrn1 | ? | 16:27 |
| *** amitgandhinz has joined #openstack-infra | 16:27 | |
| *** amitgandhinz has quit IRC | 16:27 | |
| *** mpaolino has joined #openstack-infra | 16:28 | |
| asselin | marcusvrn1, Not sure what fixed it. I adjusted the pip.conf a bit, increase timeout, and copied it to /etc/pip.conf | 16:28 |
| asselin | marcusvrn1, I found this recent patch could be a cause too, but not confirmed: https://review.openstack.org/#/c/157535/ | 16:28 |
| *** sdake__ has joined #openstack-infra | 16:29 | |
| deepakcs | fungi, ok, we are goign ahead with the rax vm, just email the thread when hpcloud vm is built, thanks | 16:29 |
| *** jcoufal has quit IRC | 16:30 | |
| fungi | deepakcs: will do--good luck | 16:30 |
| *** tmcpeak has quit IRC | 16:31 | |
| deepakcs | fungi, thanks , we need all the luck :) | 16:31 |
| *** sarob has quit IRC | 16:31 | |
| marcusvrn1 | asselin: ahh...maybe, once it was merged recently and the problem raised recently too | 16:32 |
| marcusvrn1 | asselin: another question... to configure local pip repo, do I have to change /etc/pip.conf instead of $HOME/.pip/pip.conf? | 16:32 |
| *** sdake_ has quit IRC | 16:33 | |
| *** unicell1 has quit IRC | 16:34 | |
| jlvillal | clarkb: Not sure if you can help or not. Am I doing something wrong? http://paste.openstack.org/show/181310/ My command: tox -v -v -epy34 Ubuntu 14.04 system, with updates, has Python 3.4.0. | 16:34 |
| *** markvan_ has joined #openstack-infra | 16:34 | |
| *** tmcpeak has joined #openstack-infra | 16:35 | |
| asselin | marcusvrn1, seems the /etc/pip.conf is the top level file. The others provide overrides. Seems the issue (for me) is the setting is not getting used for virtual envs. BTW, here's the error for me: http://15.126.198.151/32/158732/1/check/3par-iscsi-driver-master-client-pip-eos10-dsvm/e497b2a/logs/devstacklog.txt.gz#_2015-02-24_15_49_08_339 | 16:35 |
| clarkb | jlvillal: looks like httpretty isn't python3 friendly | 16:35 |
| *** dkranz has quit IRC | 16:36 | |
| clarkb | which is interesting because somehow the tests actually ran on the jenkins node | 16:36 |
| jlvillal | clarkb: Where would that leave me then? :) | 16:36 |
| fungi | jlvillal: yeah, new httpretty release three days ago. maybe they regressed? | 16:36 |
| *** jcoufal has joined #openstack-infra | 16:37 | |
| *** che-arne has joined #openstack-infra | 16:37 | |
| jlvillal | fungi: Hrmph. Sucks for me ;) | 16:37 |
| fungi | jlvillal: try adjusting the requirements lists for ironicclient to httpretty<0.8.7 and see if that works around it | 16:37 |
| waynr | pelix: yo, do you think we should dig into the 'submodule' option to check if it contains options corresponding to the deprecated disable-submodules and recursive-submodules options? | 16:37 |
| jlvillal | fungi: Thanks! Good idea. | 16:38 |
| *** yamahata has quit IRC | 16:38 | |
| fungi | looks like they publish a python 2.x only wheel alongside their tarball, so maybe they are acknowledging that they don't have py3k support. unfortunate indeed if so | 16:38 |
| sigmavirus24 | fungi: thanks for the ping last night. 2.5.3 is out and shouldn't cause problems. You can continue to track the issue on that requests bug you've already commented on and http://bugs.python.org/issue23476 if you're interested | 16:38 |
| fungi | sigmavirus24: yep, appreciated. i saw the updates. quick work! | 16:39 |
| krotscheck | So, I'm trying to get https://review.openstack.org/#/c/157589/ tested, and it understandably fails because it's dependent on https://review.openstack.org/#/c/157981/. The latter passes because it has the depends-on: set properly, but is there some way to make the dependee test against the depender? The email said I shouldn't create loops. | 16:39 |
| sigmavirus24 | fungi: yeah, I had a bad feeling last night but it seemed best to cut a release since there were some critical bugs it would fix | 16:39 |
| pelix | waynr: that's just it, I think that's more effort than worth it | 16:39 |
| sigmavirus24 | fungi: we're also trying to take over maintenance (specifically release maintenance) of certifi from Kenneth because of stuff like this | 16:40 |
| fungi | sigmavirus24: can't release software without breaking a few eggs | 16:40 |
| *** deepakcs has quit IRC | 16:40 | |
| *** scheuran has quit IRC | 16:40 | |
| sigmavirus24 | For what it's worth, we had to go back to our (> 1 year) old certificate which has otherwise revoked certificates in there | 16:40 |
| fungi | ick | 16:40 |
| clarkb | krotscheck: you can't because there aren't atomic multi commits. so each commit has to go in in order | 16:40 |
| sigmavirus24 | It is proving to be a hard problem to track down but it seems mostly in how OpenSSL parses/handles cross-signed certificates | 16:40 |
| sigmavirus24 | So we're in #python-requests trying to figure out ways to work around that without keeping around revoked certs | 16:41 |
| krotscheck | clarkb: So what's your suggestion there, put in a temporary migration shim so that as things land they deprecate cleanly? | 16:41 |
| sigmavirus24 | Anyone interested in helping with this should feel free to join and participate. I'll be happy to provide links for context when you get there | 16:41 |
| fungi | sigmavirus24: if i weren't preparing to head out of town i'd join you, but i'm short on time for x509 hacking | 16:41 |
| clarkb | jlvillal: fungi fwiw httpretty 0.8.7 installs fine for me using python3.4 | 16:41 |
| sigmavirus24 | fungi: fair enough =) | 16:41 |
| sigmavirus24 | fungi: safe travels | 16:41 |
| jlvillal | clarkb: Is Python 3.4.0 acceptable to use for my testing? | 16:41 |
| fungi | sigmavirus24: thanks@ | 16:41 |
| clarkb | krotscheck: ya I think you have to have a forward and backward compatible step that may end up getting cleaned up later | 16:42 |
| *** belmoreira has quit IRC | 16:42 | |
| clarkb | jlvillal: thats what tox -epy34 uses | 16:42 |
| fungi | jlvillal: should be fine (it's what we're running at the moment for python34 jobs) | 16:42 |
| krotscheck | clarkb: Ugh pain. | 16:42 |
| krotscheck | Well, at least all the other patches are ready. | 16:42 |
| *** radez_g0n3 is now known as radez | 16:42 | |
| jlvillal | clarkb, fungi: Not sure why I am failing. Thanks. | 16:42 |
| marcusvrn1 | asselin: the same here. Virtual envs are not using my pip repository... Searching for all paths to pip.conf, /etc/pip.conf was not found.. http://paste.openstack.org/show/181316/ | 16:43 |
| *** dhritishikhar has joined #openstack-infra | 16:43 | |
| waynr | pelix: okay, sounds good--thanks for the review! | 16:44 |
| *** spzala has joined #openstack-infra | 16:44 | |
| *** vigneshvar has quit IRC | 16:44 | |
| asselin | marcusvrn1, so you get the same exact error? | 16:44 |
| *** adalbas has quit IRC | 16:44 | |
| asselin | marcusvrn1, I updated my scripts to copy the pip.conf to that location, so that it gets used | 16:45 |
| jeblair | krotscheck: it's an intentional design decision to support CD (even though it would be possible in this case, simultaneous upgrades of components is not universally possible) | 16:45 |
| *** bhuvan has joined #openstack-infra | 16:46 | |
| *** amitgandhinz has joined #openstack-infra | 16:46 | |
| *** amitgandhinz has quit IRC | 16:46 | |
| *** amitgandhinz has joined #openstack-infra | 16:46 | |
| marcusvrn1 | asselin: "the same here" was about venvs are not using pip local repo. About the error in your CI, I think mine maybe something like yours, because both are not reaching the repo/packages, but it's not the same exact error | 16:47 |
| *** jcoufal has quit IRC | 16:47 | |
| asselin | marcusvrn1, I also found that file does exist in my slaves, but it belongs to -infra | 16:47 |
| *** dkranz has joined #openstack-infra | 16:47 | |
| clarkb | jlvillal also are you using latest tox? if not maybe that would help? | 16:49 |
| fungi | krotscheck: or rephrased, if it's too hard to serially implement dependent features across a set of repos, then those are arguably not suitable to keep separate | 16:49 |
| jlvillal | clarkb: I will check. I did a plain pip install and also failed for me | 16:49 |
| fungi | the repos, that is | 16:49 |
| jlvillal | clarkb: http://paste.openstack.org/show/181317/ | 16:49 |
| marcusvrn1 | asselin: I'll try to add the pip.conf file into /etc | 16:49 |
| krotscheck | fungi: I get that. I just don't like the idea of writing code that I'm going to delete later. | 16:49 |
| jlvillal | clarkb: Trying to debug... | 16:49 |
| clarkb | jlvillal it is possible that a pip or setuptools difference is at fault. tox bundles both via virtaulenv | 16:49 |
| *** tnovacik_ has quit IRC | 16:49 | |
| *** yfried has joined #openstack-infra | 16:50 | |
| jlvillal | clarkb: Okay. Thanks. My tox is the one that comes with Trusty | 16:50 |
| jlvillal | Same with pip and virtualenv | 16:50 |
| clarkb | try install latest tox in a venv and use tox from therr | 16:51 |
| fungi | krotscheck: however it serves as an example of how to manage that dependent transition in case some downstream consumer needs to do it in a non-lockstep fashion, so it's a useful habit to get into | 16:51 |
| *** yfried__ has quit IRC | 16:51 | |
| krotscheck | fungi: But whine. | 16:51 |
| jlvillal | clarkb: will do | 16:51 |
| *** yfried has quit IRC | 16:55 | |
| *** tnovacik_ has joined #openstack-infra | 16:56 | |
| *** spredzy is now known as spredzy|afk | 16:56 | |
| *** jcoufal has joined #openstack-infra | 16:57 | |
| jeblair | mrmartin: left a comment on 140043 | 16:58 |
| *** emagana has quit IRC | 16:58 | |
| *** MaxV has quit IRC | 16:59 | |
| *** emagana has joined #openstack-infra | 16:59 | |
| *** amitgandhinz has quit IRC | 16:59 | |
| clarkb | jeblair: 155773 on my other node has db problems, are you interested in that traceback? its probably because ended up installing two different HEADs that wre branches so don't share a common db migration path | 17:00 |
| mrmartin | jeblair, thnx, checking that | 17:00 |
| clarkb | jeblair: no such column: change.starred is the tl;dr error | 17:01 |
| openstackgerrit | Merged openstack-infra/infra-specs: StoryBoard integration tests specification https://review.openstack.org/150743 | 17:01 |
| jeblair | clarkb: sqlite3 .gertty.db "select * from alembic_version;" | 17:01 |
| *** mpavone has quit IRC | 17:02 | |
| *** yamahata has joined #openstack-infra | 17:02 | |
| clarkb | jeblair: 312cd5a9f878 | 17:02 |
| SpamapS | https://storyboard.openstack.org/#!/story/2000175 | 17:02 |
| *** arxcruz has quit IRC | 17:02 | |
| SpamapS | anybody want to claim "Name the cloud" ? | 17:02 |
| SpamapS | I feel like we side-stepped it. | 17:02 |
| krotscheck | SpamapS: Vanilla! | 17:03 |
| SpamapS | Not sure that's a good thing to bikeshed at the meeting. ;) | 17:03 |
| *** esker has joined #openstack-infra | 17:03 | |
| SpamapS | I was thinking we could name it Hubert, after my grandpa. ;) | 17:03 |
| *** emagana has quit IRC | 17:03 | |
| jeblair | clarkb: that's the previous version; it should have upgraded. can you run with -d and check the log? | 17:03 |
| clarkb | jeblair: sure | 17:03 |
| cinerama | who owns the ical feed with all the meeting times? | 17:04 |
| jeblair | cinerama: ttx, but if you update the wiki, he will notice and update the feed | 17:04 |
| clarkb | jeblair: http://paste.openstack.org/show/181319/ not much there | 17:05 |
| *** emagana has joined #openstack-infra | 17:05 | |
| cinerama | oh, OK, cool. i have a wishlist item of having an ical for each meeting because that is a lotta meetings | 17:05 |
| anteaya | can we delete gerrit groups or just render them deprecated and hidden? https://review.openstack.org/#/c/158666/ I know gerrit accounts are just rendered deprecated and hidden, I must admit I don't know what we do with groups we don't need | 17:05 |
| *** dhritishikhar has quit IRC | 17:05 | |
| jeblair | cinerama: ah, talk to ttx and reed about that then. they are working on automating the system | 17:06 |
| jeblair | cinerama: and see this project: http://git.openstack.org/cgit/openstack-infra/yaml2ical/tree/ | 17:06 |
| *** annegent_ has quit IRC | 17:06 | |
| clarkb | anteaya: just deprecated and hidden | 17:06 |
| jeblair | cinerama: i _think_ that might be already handled by the new tool | 17:07 |
| anteaya | clarkb: okay great, can we deprecate and hide https://review.openstack.org/#/admin/groups/573,members please? | 17:07 |
| *** ihrachyshka has quit IRC | 17:07 | |
| *** koolhead17 has joined #openstack-infra | 17:07 | |
| jeblair | clarkb: it seems like alembic doesn't see that migration file | 17:08 |
| *** _nadya_ has quit IRC | 17:08 | |
| *** dtantsur is now known as dtantsur|afk | 17:08 | |
| clarkb | jeblair: OH! maybe this is another instance of I have preexisting egg info and pip is all confusing updating | 17:08 |
| cinerama | ah ok cool. looks like we'd want to update docs and links off the wiki assuming that code DTRT | 17:08 |
| clarkb | jeblair: let me try a git clean then reinstall | 17:08 |
| marcusvrn1 | asselin: yah! adding the pip.conf into /etc, fixed the problem that venv was not using my local repo. Thanks | 17:08 |
| asselin | marcusvrn1, cool | 17:08 |
| clarkb | anteaya: we don't need that group anymore? | 17:08 |
| * clarkb tries to remember what suffix we were renaming deprecated groups too | 17:08 | |
| clarkb | s/too/to/ | 17:09 |
| anteaya | clarkb: no ttx doesn't want it as he wants to use the existing tech-committee-chair group: https://review.openstack.org/#/c/158666/ | 17:09 |
| clarkb | jeblair: yup that fixed it, removed the old gertty egg stuff | 17:09 |
| *** annegent_ has joined #openstack-infra | 17:10 | |
| clarkb | anteaya: ok we should let that merge and get applied before we rename tc-chair and make it invisible | 17:10 |
| anteaya | clarkb: fair enough | 17:10 |
| * SpamapS decides we'll just have to open the bikeshed up in the meeting. :-P | 17:10 | |
| anteaya | thanks | 17:10 |
| *** vigneshvar has joined #openstack-infra | 17:10 | |
| *** mpaolino has quit IRC | 17:11 | |
| *** tmcpeak has quit IRC | 17:11 | |
| sdague | clarkb: hmmm... gerrit just gave me a 500 on trying to go to the next page of a review | 17:11 |
| clarkb | anteaya: np remind me once that gets in and I will turn the gerrit knobs | 17:11 |
| jeblair | krotscheck, yolanda: does the following command run in the foreground? tox -e venv "storyboard-api --config-file ./etc/storyboard.conf.test" | 17:11 |
| sdague | did it ever get a restart? | 17:11 |
| clarkb | sdague: it did on friday | 17:12 |
| yolanda | yes, it works for me, is the one i normally use | 17:12 |
| sdague | clarkb: ok | 17:12 |
| clarkb | sdague: http://i.imgur.com/QnFgwGV.png was the result | 17:12 |
| jeblair | yolanda: right, but it runs in the foreground, yes? it does not start a daemon in the background? | 17:12 |
| krotscheck | jeblair: I think so, yolanda will be able to give you more detail. | 17:12 |
| clarkb | I will check that graph again | 17:12 |
| sdague | clarkb: does it log anything sane on a 500? | 17:12 |
| yolanda | yes, no daemon | 17:12 |
| *** amotoki has quit IRC | 17:12 | |
| anteaya | clarkb: it has some errors so I will look at 158666 and let you know | 17:12 |
| jeblair | yolanda: cool, thanks. | 17:12 |
| mrmartin | jeblair: all other files in this directory have jetty:jetty ownership, as provided by vamsee-solr plugin. | 17:13 |
| clarkb | sdague: usually not but I can check that too (giant java exceptins usually) | 17:13 |
| clarkb | sdague: what change is it? | 17:13 |
| sdague | https://review.openstack.org/#/c/138956/8/doc/source/guides/nova.rst,cm | 17:13 |
| yolanda | jeblair, btw, saw the approval for the spec, thx | 17:13 |
| jeblair | mrmartin: okay, then i'm in favor of consistency. i'll change to +2. i do think that's weird though. | 17:13 |
| clarkb | memory use is still well under the 8GB heap limit so we aren't bumping against that | 17:13 |
| mrmartin | jeblair: we can deliver an upstream patch for this solr module | 17:14 |
| anteaya | well the gate jobs failed and have no logs: https://review.openstack.org/#/c/158666/ | 17:14 |
| *** tmcpeak has joined #openstack-infra | 17:14 | |
| jeblair | mrmartin: might be a good idea | 17:15 |
| anteaya | I can recheck | 17:15 |
| mrmartin | need to something anyway, becuase it not works well with 4.10.x solr | 17:15 |
| clarkb | sdague: 1) Cannot open ReviewDb is the gerrit error | 17:17 |
| sdague | clarkb: yeh, the 500 was surprising | 17:17 |
| sdague | clarkb: interesting... db contention? | 17:17 |
| clarkb | Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure | 17:17 |
| clarkb | looks like connection problems but reading the mass of tracebacks now | 17:18 |
| mrmartin | jeblair: may I get a +2 for this too: https://review.openstack.org/#/c/152508/ ? | 17:18 |
| *** jistr has quit IRC | 17:18 | |
| clarkb | Connection timed out | 17:18 |
| clarkb | zaro: ^ is that what you were seeing on review-dev? | 17:18 |
| *** carl_baldwin has quit IRC | 17:18 | |
| clarkb | fungi: ^ did you update review.o.o's db config too? | 17:18 |
| *** rwsu-afk is now known as rwsu | 17:18 | |
| *** erlon has joined #openstack-infra | 17:19 | |
| *** emagana has quit IRC | 17:19 | |
| *** coolsvap_ is now known as coolsvap | 17:20 | |
| clarkb | anteaya: https://jenkins07.openstack.org/job/gate-infra-puppet-apply-trusty/334/console we are being bit by the requests problem | 17:21 |
| *** dangers is now known as dangers_away | 17:22 | |
| *** hdd has joined #openstack-infra | 17:22 | |
| reed | cinerama, have you looked at the code on http://git.openstack.org/cgit/openstack-infra/yaml2ical/tree ? | 17:22 |
| anteaya | clarkb: thanks | 17:22 |
| *** tmcpeak has quit IRC | 17:22 | |
| anteaya | I wondered if it was something bigger | 17:22 |
| *** matrohon has quit IRC | 17:23 | |
| anteaya | but I had no logs returned at all | 17:23 |
| cinerama | reed: i just did (v briefly) after jeblair mentioned it | 17:23 |
| clarkb | sdague: my initial theory is that we hit the 2 minute connection timeout that trove gives us by default, but need to double check that fungi that new config was applied and mysql reloaded its config | 17:23 |
| sdague | clarkb: ok, coolio. Not a huge deal, but figured it was worth reporting | 17:24 |
| sdague | thanks for looking into it | 17:24 |
| anteaya | found a log on a running job and yeah, that is it | 17:24 |
| *** dhritishikhar has joined #openstack-infra | 17:24 | |
| *** tonytan4ever has joined #openstack-infra | 17:24 | |
| clarkb | sdague: np, and yes, knowing when gerrit is broken is super handy :) | 17:25 |
| openstackgerrit | Merged openstack-infra/infra-manual: For openstack projects, add dependency on governance change https://review.openstack.org/156745 | 17:25 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new caching class for shade https://review.openstack.org/158138 | 17:26 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/devstack-gate: Fix tox warnings for indent not multiples of 4 https://review.openstack.org/158786 | 17:26 |
| *** afazekas has quit IRC | 17:26 | |
| *** vilobhmm has joined #openstack-infra | 17:27 | |
| jlvillal | clarkb: Argh, that took awhile. So my issue was that I have LANG=C in my environment. Once I let it default to: LANG=en_US.UTF-8 then httpretty installed. | 17:27 |
| *** viktors is now known as viktors|afk | 17:27 | |
| zaro | clarkb, fungi : i do not have powers to change the global timeout for review-dev.o.o db | 17:28 |
| *** emagana has joined #openstack-infra | 17:28 | |
| *** afazekas has joined #openstack-infra | 17:28 | |
| zaro | or at least i don't think i do. anyways i thought fungi changed that thru some UI? | 17:28 |
| clarkb | zaro: I know, I am asking if this is the same issue | 17:29 |
| *** baoli has quit IRC | 17:29 | |
| zaro | yeah, review-dev.o.o will continue to timeout due to inactivity. you don't see it on review.o.o because it's always active. | 17:30 |
| clarkb | zaro: right but I just saw a connection timeout out review.o.o | 17:30 |
| anteaya | jlvillal: thanks for sharing the solution | 17:30 |
| *** Ala has quit IRC | 17:30 | |
| *** tiswanso has quit IRC | 17:30 | |
| clarkb | so my question is, does the error message above look like what you were seeing on review-dev? | 17:30 |
| *** skolekonov has quit IRC | 17:30 | |
| *** baoli has joined #openstack-infra | 17:30 | |
| zaro | ohh, yeah. it's unlikely but possible isn't it? | 17:31 |
| jlvillal | anteaya: Filing a bug with httpretty on it | 17:31 |
| zaro | clarkb: but yeah, that looks like the same error | 17:31 |
| *** tiswanso_ has joined #openstack-infra | 17:31 | |
| clarkb | zaro: thanks | 17:31 |
| anteaya | jlvillal: okay | 17:33 |
| *** amrith has left #openstack-infra | 17:33 | |
| *** femto has joined #openstack-infra | 17:33 | |
| reed | cinerama, would that solve your issues? | 17:35 |
| *** unicell has joined #openstack-infra | 17:36 | |
| *** marun has joined #openstack-infra | 17:36 | |
| cinerama | reed: absolutely. having per-meeting ics available is what i'm after | 17:36 |
| *** tnurlygayanov has quit IRC | 17:37 | |
| anteaya | jlvillal: learned something new about LANG=C, thanks | 17:37 |
| *** femto has quit IRC | 17:37 | |
| *** annegent_ has quit IRC | 17:37 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/requirements: Add ryu>=3.18 https://review.openstack.org/154354 | 17:38 |
| *** adalbas has joined #openstack-infra | 17:38 | |
| fungi | clarkb: sounds reasonable. fwiw the config profile is available within that tenant and you can apply it to any trove instance you want. should just be able to select it from the config drop-down on the instance. i believe i named it something appropriately obvious but forget exactly what | 17:39 |
| *** rhe00_ has joined #openstack-infra | 17:39 | |
| * anteaya pictures clark looking for the option "appropriately obvious" | 17:40 | |
| rhe00_ | how can I turn off the debug prints from nodepool? | 17:40 |
| *** rhe00_ is now known as rhe00 | 17:40 | |
| clarkb | fungi: cool, do you know if I have to manual trigger a config reload in the instance? and if that causes connections to drop in order to restart the server? | 17:40 |
| clarkb | rhe00: you want to disable the debug.log file? | 17:40 |
| *** tonytan4ever has quit IRC | 17:40 | |
| fungi | clarkb: i don't think i had to do anything to make it take effect | 17:41 |
| fungi | clarkb: other than just select it | 17:41 |
| clarkb | mordred: you are our resident mysql person, any thoughts on my applying our review-dev config profile to review.o.o? | 17:41 |
| * fungi is mostly afk at the moment preparing to head out | 17:41 | |
| *** ghostpl_ has quit IRC | 17:41 | |
| mordred | hm | 17:41 |
| clarkb | mordred: would be done to avoid the db connection timeouts that we were seeing, sdague ran into one today on review.o.o | 17:41 |
| mordred | clarkb: this is the increate-of-timeout setting, yeah? | 17:42 |
| rhe00 | clarkb: I want to disable the prints marked as DEBUG coming out on the screen after I execute "nodepoold -d". | 17:42 |
| clarkb | mordred: ya the 2 minute bomb countdown from every action movie ever | 17:43 |
| * SpamapS pokes at gertty to try and make the diff added lines not-quite-as-dark green. | 17:43 | |
| clarkb | rhe00: you can pass it a log config with the -l argument and in that log config you can disable debug output | 17:43 |
| mordred | clarkb: sounds fine to me | 17:43 |
| *** ghostpl_ has joined #openstack-infra | 17:44 | |
| *** derekh has quit IRC | 17:44 | |
| mrmartin | this failed on gate-ci-docs: https://review.openstack.org/#/c/152508/ and cannot find the logs: http://logs.openstack.org/08/152508/3/gate/gate-ci-docs/aa657ed/ | 17:44 |
| clarkb | rhe00: https://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/templates/nodepool.logging.conf.erb that is our logging config which does keep debug logs | 17:44 |
| jeblair | clarkb, mordred: ++ | 17:44 |
| clarkb | cool I will go ahead and figure out how to apply that profile now | 17:45 |
| rhe00 | clarkb: I edited /etc/nodepool/logging.conf and removed all references to debug, but that did not seem to help | 17:46 |
| anteaya | mrmartin: yes this is a known problem | 17:46 |
| anteaya | mrmartin: there is a cert issue | 17:46 |
| *** dizquierdo has quit IRC | 17:46 | |
| clarkb | rhe00: did you pass the path to that file with -l? | 17:46 |
| *** Ryan_Lane has joined #openstack-infra | 17:46 | |
| mrmartin | anteaya: is it a permanent one, or just something that will go away? | 17:46 |
| clarkb | rhe00: if you don't the default behavior is to log at debug level | 17:46 |
| anteaya | well it won't go away until it is fixed | 17:47 |
| jeblair | anteaya: oh, the requests library cert issue is affecting swift log uploads? | 17:47 |
| clarkb | jeblair: yes | 17:47 |
| jeblair | neato | 17:47 |
| anteaya | yes | 17:47 |
| anteaya | error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed | 17:47 |
| anteaya | https://jenkins06.openstack.org/job/gate-infra-puppet-apply-precise/922/console | 17:47 |
| *** carl_baldwin has joined #openstack-infra | 17:47 | |
| rhe00 | clarkb: I did not pass in the path. I though it would look at that file by default. I will try the -l option next. Thanks! | 17:48 |
| jroll | fungi: and friends, requests 2.5.3 appears to fix the rackspace identity ssl problems | 17:49 |
| anteaya | jroll: yay | 17:49 |
| zaro | clarkb: the gerrit error was more like.. Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: | 17:49 |
| zaro | The last packet successfully received from the server was 216,796,727 | 17:49 |
| zaro | milliseconds ago | 17:49 |
| zaro | clarkb: https://groups.google.com/forum/#!msg/repo-discuss/rR_ouEeAQzU/bA70CYLT0t0J | 17:50 |
| clarkb | zaro: ya that looks right | 17:50 |
| clarkb | anteaya: we will need to rebuild images | 17:50 |
| anteaya | clarkb: yay | 17:51 |
| pleia2 | goodie | 17:51 |
| * anteaya continues to not approve anything right now | 17:51 | |
| clarkb | pleia2: any chance you want to rebuild images :) | 17:51 |
| clarkb | pleia2: I can walk you through it | 17:51 |
| anteaya | pleia2: :) | 17:51 |
| anteaya | go you | 17:51 |
| *** jgjl has quit IRC | 17:52 | |
| pleia2 | clarkb: sure | 17:52 |
| rcarrillocruz | mordred: just in case it's not under your radar, you may want to check https://review.openstack.org/#/c/158615/ | 17:52 |
| rcarrillocruz | (hi) | 17:52 |
| * anteaya follows along | 17:52 | |
| clarkb | pleia2: I like to hop on nodepool.o.o then start a screen session because I move around enough and builds take long enough that otherwise badness happens | 17:52 |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:52 | |
| clarkb | pleia2: then instead of using the all shortcut to build images for all the regions that use an image I prefer to build a few at a time, its slightly more work but I get builds in parallel and its easier to follow along | 17:53 |
| pleia2 | clarkb: sounds good, I've logged on | 17:53 |
| clarkb | pleia2: so for that its `sudo -H -u nodepool nodepool image-update $region $image-name`. That mostly works for everything except for dib | 17:53 |
| *** ParsectiX has joined #openstack-infra | 17:54 | |
| pleia2 | clarkb: ok, so where do we want to start region/image-wise? | 17:54 |
| clarkb | pleia2: a good place to start is probably `sudo -H -u nodepool nodepool rax-dfw bare-trusty` and rax-ord, rax-iad, hpcloud-b1 through hpcloud-b5 because thats not dib | 17:54 |
| clarkb | pleia2: the reason I would start with bare-trusty is we use more trusty nodes than any other releases at this point and the bare-* don't use dib yet so you can fire those off while I go over the small exception to the dib build process | 17:55 |
| arosen | Hi, I was curious if zuul or jenkins has any way to prioritize which patches in the queue are run first? | 17:55 |
| arosen | For example, say you want to give priority to openstack commits vs stackforge commits. | 17:55 |
| pleia2 | clarkb: ok, and we get the list of regions and images from nodepool.yaml.erb? | 17:56 |
| *** cody-somerville has quit IRC | 17:56 | |
| clarkb | pleia2: yup, also `sudo -H -u nodepool nodepool image-list` shows you what we alread have | 17:56 |
| jeblair | arosen: only by pipeline, not by project. | 17:56 |
| pleia2 | clarkb: ah, cool | 17:56 |
| clarkb | pleia2: I like to pipe that command through grep on a region name or an image name typically | 17:56 |
| jeblair | arosen: we thought about that, but every time we look, stackforge continues to be a tiny portion of the load, so it's not worth doing. | 17:57 |
| arosen | jeblair: cool interesting to know! Thanks. | 17:57 |
| pleia2 | clarkb: alright, shall I begin with rax-dfw bare-trusty then? | 17:57 |
| reed | is anybody aware of a way to reply to openstack-dev messages via web? I noticed gossamer-threads and gmane both are read-only for our list | 17:57 |
| clarkb | pleia2: yup, I think you can do the whole set for bare-trusty | 17:57 |
| jeblair | reed: gmail? :) | 17:57 |
| clarkb | rax-dfw, rax-iad, rax-ord, hpcloud-b1, hpcloud-b2, hpcloud-b3, hpcloud-b4, hpcloud-b5 | 17:58 |
| pleia2 | clarkb: but suggesting doing them one by one? | 17:58 |
| reed | jeblair, :) right ... the use case is of a person who subscribed to the (quite useless IMHO) digest and wants to reply without breaking the thread | 17:58 |
| clarkb | my personal preference is to do one per screen window foregroudned so I can follow along | 17:58 |
| clarkb | pleia2: but thats how my brain works, I think fungi ends up scripting it a bit more with for loops and subshells | 17:58 |
| pleia2 | aha :) | 17:59 |
| clarkb | pleia2: should be fine to start all of them at once for the bare-trusty image | 17:59 |
| pleia2 | maybe I'll start with this one then do the rest at once | 17:59 |
| clarkb | sounds good | 17:59 |
| bcourt | hi, has there been any though regarding having a rest api for working with nodepool? Having a way to query the information & make updates without being on the local box would be helpful | 17:59 |
| *** tnovacik_ has quit IRC | 17:59 | |
| clarkb | bcourt: rcarrillocruz has a change to add an api to it | 18:00 |
| *** lucasagomes is now known as lucas-dinner | 18:00 | |
| *** lucas-dinner has left #openstack-infra | 18:00 | |
| jeblair | reed: i don't use mm digests, but if they are MIME digests, then individual messages should be able to be opened and then replied to | 18:00 |
| * bcourt goes to find that change | 18:00 | |
| clarkb | my concern right now is that nodepool runs a bit privileged for dib and I wouldn't want to expose an api without a lot of care | 18:00 |
| rcarrillocruz | bcourt: https://review.openstack.org/#/c/136884/ | 18:00 |
| *** ihrachyshka has joined #openstack-infra | 18:00 | |
| *** coolsvap is now known as coolsvap_ | 18:00 | |
| clarkb | pleia2: let me know when you have th efirst one running and I can go over the small dib exception | 18:00 |
| *** bdpayne has joined #openstack-infra | 18:01 | |
| pleia2 | clarkb: it's running | 18:01 |
| *** carl_baldwin has quit IRC | 18:01 | |
| *** annegent_ has joined #openstack-infra | 18:01 | |
| reed | jeblair, I'll look more into this... last time I checked, the digest was one large txt file, unreadable | 18:01 |
| jeblair | reed: also, the person could ask a friend (you? :) to bounce (resend) them a copy of the message | 18:01 |
| clarkb | pleia2: great, so the things that makes dib images slightly special is that if yuo do `nodepool image-update someregion dib-image` what that does is first build a new dib image then upload it, however what we ideally want is to build a single dib image then upload that to all of our regions that can use the dib image | 18:02 |
| reed | jeblair, AH! that's a good idea, thanks! | 18:02 |
| reed | a quick workaround | 18:02 |
| pleia2 | clarkb: makes sense | 18:02 |
| rcarrillocruz | jeblair: re: devstack for shade, i know, but i'm not sure how to split my tests into a job that is not run on python-jobs template, i.e. https://review.openstack.org/#/c/158463/ | 18:02 |
| rcarrillocruz | could you pls advise? | 18:02 |
| clarkb | pleia2: so my preference for dib based images, in this case devstack-* against hpcloud-* only os the do a `sudo -H -u nodepool nodepool image-build devstack-trusty` which only does the build then do a `sudo -H -u nodepool nodepool image-upload hpcloud-b1 devstack-trusty` | 18:03 |
| clarkb | pleia2: then do that image-upload step for each dib'd region | 18:03 |
| pleia2 | clarkb: gotcha | 18:03 |
| jeblair | rcarrillocruz: oh, do you not have a job defined for that yet? | 18:03 |
| clarkb | pleia2: if you end up forgetting and just using image-update its not the end of the world but preferable to do a single image build then multiple uploads of that image | 18:03 |
| * pleia2 nods | 18:04 | |
| *** nfedotov has quit IRC | 18:04 | |
| *** jcoufal has quit IRC | 18:04 | |
| clarkb | (nothing should break) | 18:04 |
| *** jcoufal has joined #openstack-infra | 18:04 | |
| *** dhritishikhar has quit IRC | 18:04 | |
| rcarrillocruz | no... thing is, python-jobs runs whatever it finds in repo, i'm not sure how to prevent that... | 18:04 |
| *** bhuvan has quit IRC | 18:04 | |
| jlvillal | clarkb: jroll: So the python-ironicclient py34 test seems to be hanging at: subunit.run discover Any debug ideas? I'm doing this on tip of tree of python-ironicclient | 18:05 |
| *** markus_z has quit IRC | 18:05 | |
| clarkb | pleia2: and finally we do the whole sudo -H -u nodepool thing so that file permissions stay consistent with what the running daemon wants | 18:05 |
| *** ssam2 has quit IRC | 18:05 | |
| pleia2 | clarkb: yeah, using the nodepool user to do it all | 18:05 |
| *** jp_at_hp1 has quit IRC | 18:05 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Bump tempest-lib min version to 0.3.0 https://review.openstack.org/158803 | 18:05 |
| jeblair | rcarrillocruz: okay, a couple of things you need to do: | 18:06 |
| pleia2 | clarkb: once the image is built, do we do any tests to make sure it's "right"? | 18:06 |
| jeblair | rcarrillocruz: in shade, make the standard tox env only run unit tests. edit tox.ini to change the command to specify which tests to run. | 18:06 |
| * rcarrillocruz reads | 18:06 | |
| *** kmartin has quit IRC | 18:06 | |
| rcarrillocruz | oh, at the tox level | 18:06 |
| jeblair | rcarrillocruz: in shade, add a new tox env to tox.ini named "func" which runs the functional tests | 18:06 |
| jroll | jlvillal: I have no idea without diving in :x | 18:06 |
| clarkb | pleia2: no, we have per VM boot ready scripts that we run that do a quick check but beyond that we wan't errors like this to bubble up where people see them rather than scrubbing images that fail hard and continuing to use ancient images instead | 18:06 |
| rcarrillocruz | gotcha | 18:07 |
| jeblair | rcarrillocruz: then in project-config, add a new job that runs the functonal tox tests on a devstack node | 18:07 |
| jeblair | rcarrillocruz: you can probably look at swift as a model for how to do all of that | 18:07 |
| rcarrillocruz | sweet, i'll have a look | 18:07 |
| rcarrillocruz | thx a ton jeblair | 18:07 |
| jlvillal | clarkb: jroll: I can see utilization 96.7% running for 22:32.39 'python -m subunit.run discover -t ./ ./ --load-list /tmp/tmp5ixu0v69' | 18:07 |
| *** armax has quit IRC | 18:07 | |
| jeblair | np! | 18:07 |
| pleia2 | clarkb: ok, thanks | 18:07 |
| jroll | jlvillal: yeah, something is clearly borked but idk what... link to patch if you don't mind? | 18:07 |
| clarkb | jlvillal: discover is how it finds the list of tests, it imports all the things below your specified test root and finds things that match the test rules. My guess is that you have code that executes at import time somewher and py34 is making it sad | 18:07 |
| jlvillal | clarkb: jroll: I'm unfamiliar with testr, so not really sure how to proceed | 18:07 |
| jlvillal | jroll: This is tip of tree | 18:08 |
| clarkb | jlvillal: in this case its not even testr or subunit, its part of the python testing stdlib bits | 18:08 |
| jroll | oh good | 18:08 |
| *** doude has joined #openstack-infra | 18:08 | |
| jroll | x.x | 18:08 |
| clarkb | bcourt: out of curiousity what specific things are you needing to do with nodepool where logging into the machine isn't practical? | 18:08 |
| rcarrillocruz | clarkb: ideally, i'd like to have the Nodepool REST API with some authorization layer, with groups and stuff... maybe there's no use case for such granularity on Infra, but in downstream it would make a lot of sense. My change does not have all of that because I thought of coding it with Infra in mind | 18:09 |
| *** prad has quit IRC | 18:09 | |
| *** HeOS has quit IRC | 18:09 | |
| *** jcoufal has quit IRC | 18:09 | |
| rcarrillocruz | i.e. you infra-root having access as admin and that it | 18:09 |
| clarkb | rcarrillocruz: I guess, from my perspective if nodepool isn't automagically doing things 99% of that time thats probably a bug and an api doesn't fix that bug | 18:09 |
| *** AJaeger_ has joined #openstack-infra | 18:10 | |
| clarkb | rcarrillocruz: for the exceptional case of requests fixed a bug that affects us so we want to kick off new builds a human has to get involved anways so I don't mind using ssh to do that | 18:10 |
| mordred | I think having a REST API would be great | 18:10 |
| mordred | because it would allow us to build other automation around things | 18:10 |
| clarkb | mordred: ya I don't have anything gainst it other than the poential security issues, just wondering what people intend on doing with it | 18:11 |
| mordred | nod | 18:11 |
| *** devlaps has joined #openstack-infra | 18:11 | |
| jeblair | clarkb: i wondered the same thing; though we did identify a potential use last week in quiescing providers | 18:11 |
| clarkb | (also the security concerns can be addressed and there is a spec that would do so as far as dib building goes) | 18:11 |
| *** baoli has quit IRC | 18:11 | |
| bcourt | clarkb, it's a nice to have more than a need to have. Having a rest api would make it convenient to tie into an external monitoring service. It would also be somewhat useful for performing batch operations, though that could be made easier by having a flag fro the CLI to make it more text parser friendly | 18:11 |
| *** cnesa has joined #openstack-infra | 18:11 | |
| bcourt | also, given a rest API I could mash together a simple web page for folks to be able to see the current state without having to provide access to the server itself | 18:12 |
| clarkb | bcourt: what state is that? a listing of images and nodes? | 18:12 |
| bcourt | yup, current images, current nodes, start & monitor an image update | 18:13 |
| clarkb | jeblair: I'm still not sure that quiescing providers requires an api :) | 18:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Rename functional files to projects. https://review.openstack.org/158807 | 18:13 |
| jeblair | clarkb: i'm not either; it's a potential use. :) | 18:14 |
| clarkb | jeblair: it certainly doesn't hurt. Part of my concern here is I have spent a lot of time recently fixing all the bugs I can in nodepool and from my perspective thats infinitely more valuable right now than add rest api, but I am biased :) | 18:14 |
| clarkb | basically I think focus should be on fixing issues that make people think they need this extra access rather than just giving it out | 18:14 |
| *** Longgeek has quit IRC | 18:15 | |
| *** vigneshvar has quit IRC | 18:15 | |
| jeblair | clarkb: i agree it is not a high priority, particularly not a higher one than bug fixing. | 18:15 |
| *** marcusvrn1 has quit IRC | 18:16 | |
| *** adalbas has quit IRC | 18:16 | |
| *** cnesa has quit IRC | 18:16 | |
| *** otter768 has joined #openstack-infra | 18:16 | |
| jeblair | clarkb: and yeah, if the goal is to manually fix where nodepool messed something up, effort is better spent elsewhere. | 18:16 |
| bcourt | clarkb, I would generally agree, by and large nodepool has been a huge help for me. The biggest area I keep thinking about working on a patch for is adding support for sub-nodes that are of different types than the primary | 18:16 |
| *** baoli_ has joined #openstack-infra | 18:16 | |
| clarkb | jeblair: gertty threading, any thoughts on how gertty might grok depends-on to thread those changes too? | 18:16 |
| jeblair | clarkb: i think there might be some potential cases around quiescing, or kicking off image builds, or querying status (though that could be something like status.json) where it could be useful. | 18:17 |
| *** changbl has quit IRC | 18:17 | |
| jeblair | clarkb: hrm. most gertty views are single-project, so it shouldn't generally be necessary... | 18:18 |
| *** marcusvrn1 has joined #openstack-infra | 18:18 | |
| jeblair | clarkb: but in a multi-project view, i suppose we could parse the commit message and match change-ids | 18:18 |
| AJaeger_ | What's up with the gate? I see jobs failing in the gate queue that past the check queue - and then there are no log files available ;( | 18:18 |
| anteaya | AJaeger_: the swift log uploads are broken right now | 18:19 |
| anteaya | a python requests lib was causing issues with certs verification to rax | 18:19 |
| anteaya | the latest request release fixes it and pleia2 is updating the image builds | 18:19 |
| AJaeger_ | anteaya: Thanks for the info ;( So, no sense in approving anything right now, I guess? | 18:19 |
| anteaya | AJaeger_: I'm having a tea and reading things | 18:19 |
| AJaeger_ | ;) | 18:19 |
| clarkb | jeblair: ya mostly just thinking how it may work, the project centric reviewing may make it more work than its worth though | 18:19 |
| anteaya | AJaeger_: not much unless you wnat to do it again real soon | 18:20 |
| *** bhuvan has joined #openstack-infra | 18:20 | |
| *** harlowja has joined #openstack-infra | 18:20 | |
| *** ChuckC has quit IRC | 18:20 | |
| *** patrickeast has joined #openstack-infra | 18:20 | |
| *** ChuckC has joined #openstack-infra | 18:21 | |
| * mordred wants non-project-centric viewing in gertty - hasn't gotten around to trying the custom dashboard for that | 18:21 | |
| *** otter768 has quit IRC | 18:21 | |
| clarkb | mordred: SpamapS greghaynes can you see my comment on 158413? curious about how dib will handle that | 18:21 |
| jeblair | AJaeger_: only projects using swift uploads are broken (eg project-config and other infra repos) | 18:21 |
| *** annegent_ has quit IRC | 18:22 | |
| stevemar | bump, could i get eyes on https://review.openstack.org/#/c/158045/ | 18:22 |
| *** harlowja has quit IRC | 18:24 | |
| pleia2 | clarkb: it's doing a lot of "2015-02-24 18:23:47,795 INFO urllib3.connectionpool: Starting new HTTPS connection (1): dfw.servers.api.rackspacecloud.com | 18:24 |
| pleia2 | normal? | 18:24 |
| *** harlowja has joined #openstack-infra | 18:24 | |
| clarkb | pleia2: ya, it hits the api endpoints a few times iirc | 18:24 |
| clarkb | pleia2: its polling for the node to transition from building to ready iirc, then it starts attempling to ssh in | 18:24 |
| mordred | clarkb: that'll be a greghaynes question | 18:24 |
| clarkb | attempling is a new word I have just invented | 18:25 |
| clarkb | mordred: thanks, I will bug him | 18:25 |
| clarkb | greghaynes: you have been bugged, please see my comment on 158413 and enlighten my noobness | 18:26 |
| sdague | pleia2: fwiw, we turn the logging level to WARN for those libraries in oslo.log for that reason. It's a completely gratuitous message that confuses people | 18:26 |
| *** changbl has joined #openstack-infra | 18:27 | |
| clarkb | sdague: we are just using novaclient, does it not oslo.log properly? or I wonder if our logging config is conflicting with oslo.log | 18:27 |
| clarkb | eg we use normal python logging which may undermine oslo.loggings attempts at being smart for us | 18:28 |
| *** marcusvrn has joined #openstack-infra | 18:28 | |
| *** sergsh has quit IRC | 18:28 | |
| sdague | yeh, if you take over python logging, I think not of our overrides apply | 18:28 |
| *** adalbas has joined #openstack-infra | 18:29 | |
| sdague | s/not/none/ | 18:29 |
| *** marcusvrn1 has quit IRC | 18:29 | |
| clarkb | thats really unfortunate, lots of python uses logging straight and oslo.logging shouldn't assume it is the only thing importing logging | 18:29 |
| clarkb | any idea if that is a bug (I can file it) or a design decision for sanity? | 18:29 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:30 | |
| sdague | well, I'm not sure what the correct behavior is, because if you are specifying global overrides for logging it seems like you are specifying them for a reason | 18:30 |
| pleia2 | ok, dfw bare-trusty built, just kicked off builds for the rest | 18:31 |
| *** gyee has joined #openstack-infra | 18:31 | |
| greghaynes | clarkb: ugh | 18:32 |
| clarkb | sdague: good point, inherit vs override isn't necessarily always going to be correct depending on situation and the conservative override may be easiest for most to deal with | 18:32 |
| clarkb | greghaynes: yes I think this exposes a bug in how dib does grub | 18:32 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Setup non-voting VPNaaS func test for StrongSwan https://review.openstack.org/158389 | 18:32 |
| clarkb | greghaynes: it really needs to separate the install from the config | 18:32 |
| clarkb | greghaynes: that could be in finalize if necessary but you need to be able to slip other bits in between | 18:33 |
| anteaya | I just saw a turkey fly past my window | 18:33 |
| anteaya | never seen one on the wing before | 18:33 |
| anteaya | very long glide | 18:33 |
| pleia2 | clarkb: so doing all bare-trusty, need to do bare-precise, bare-centos6, any of the devstack ones? | 18:33 |
| anteaya | beautiful | 18:33 |
| fungi | okay, i'm settled into the car and caught up on scrollback. all emergencies seem to already be well in hand but let me know if assistance is required | 18:33 |
| pleia2 | anteaya: scary :) | 18:33 |
| anteaya | pleia2: majestic | 18:33 |
| timrc | Is it me or does this not actually tell you how to write a test for testr? https://wiki.openstack.org/wiki/Testr#Writing_Tests_for_testr | 18:33 |
| *** baoli_ has quit IRC | 18:33 | |
| fungi | timrc: that sounds like an existential observation | 18:34 |
| pleia2 | clarkb: since it's just the systems with swift logs, so only ones infra uses | 18:34 |
| clarkb | pleia2: ya we will need to do devstack-trusty and devstack-precise but we can get the bare-* images done first | 18:34 |
| greghaynes | clarkb: yep, definitely. I wonder if we should just install grub at start of finalise.d and configure it at the end | 18:34 |
| clarkb | pleia2: two of the devstack-gate jobs use swift logs too | 18:34 |
| sdague | clarkb: yeh, though long term I'd actually like to make oslo.log not be just a thin python logging wrapper, but more of a log message dispatcher with custom filtering in it's layer (so it could be disjoint from global python logging configs) . But it's a longer design process. | 18:34 |
| greghaynes | although, that really begs the question of why were not just doing install of grub at the start of install.d like everything else | 18:34 |
| clarkb | greghaynes: +1 | 18:34 |
| greghaynes | clarkb: ill try moving it into install.d and see what blows up | 18:34 |
| fungi | clarkb: did you by any chance get my restrict-memory fix merged before kicking off image updates? | 18:34 |
| clarkb | fungi: no because I -1'd it | 18:35 |
| *** yginger5 is now known as yjiang5 | 18:35 | |
| clarkb | fungi: see my comment I don't think it will fix the problem and greghaynes seems to confirm | 18:35 |
| fungi | oh, poop. i'll look again. you had +2 on it before | 18:35 |
| clarkb | fungi: its close but not sufficient | 18:35 |
| fungi | k | 18:35 |
| notmyname | jeblair: sdague: based on https://review.openstack.org/#/c/158671/ it might be worthwhile checking other jobs that may have had a 'sudo' added | 18:35 |
| *** devvesa has joined #openstack-infra | 18:35 | |
| clarkb | ya I must've had a dream that told me to -1 it for that reason or something :) | 18:35 |
| fungi | heh | 18:35 |
| *** adalbas has quit IRC | 18:37 | |
| *** ParsectiX has quit IRC | 18:37 | |
| clarkb | greghaynes: danke | 18:37 |
| *** yfried has joined #openstack-infra | 18:38 | |
| *** ParsectiX has joined #openstack-infra | 18:38 | |
| anteaya | sdague: you offered the orignial patch this patch modifies, care to offer an opinion? https://review.openstack.org/#/c/158671/1 | 18:39 |
| *** ivar-lazzaro has joined #openstack-infra | 18:39 | |
| sdague | yeh, seems fine | 18:40 |
| clarkb | gah haven't done the trove thing yet, I will wait for after the meeting so that I don't have to split focus | 18:40 |
| anteaya | sdague: thanks | 18:40 |
| *** rfolco has quit IRC | 18:40 | |
| sdague | notmyname: that and the swift one were the only ones added | 18:40 |
| clarkb | sdague: notmyname might make sense to have two different tox targets or have the single tox target run both for us? (at least then swift has control over how its run and doesn't need job updates for things like that) | 18:41 |
| sdague | notmyname: thanks for the fix | 18:41 |
| notmyname | it was all acoles. he noticed and fixed it. I just mentioned it in here | 18:41 |
| *** ParsectiX has quit IRC | 18:41 | |
| sdague | clarkb: yeh, probably, though this was an attempt at minimal change from what existed | 18:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add support for volume attach/detach https://review.openstack.org/153623 | 18:41 |
| *** ParsectiX has joined #openstack-infra | 18:41 | |
| notmyname | clarkb: ya, 2 jobs would be fine I think. also they'd run in parallel, I think (right?) | 18:42 |
| *** baoli has joined #openstack-infra | 18:42 | |
| clarkb | notmyname: no not two jobs | 18:42 |
| notmyname | oh 2 tox targets | 18:42 |
| *** cody-somerville has joined #openstack-infra | 18:42 | |
| clarkb | notmyname: instead update your tox.ini to either have two targets or run both sets of tests under one target | 18:42 |
| pleia2 | clarkb: getting errors on some of these builds, and when I look at the age in the list it's like 555.86 - so been broken for quite some time? | 18:43 |
| pleia2 | clarkb: Exception: Unable to run setup script | 18:43 |
| *** sputnik13 has joined #openstack-infra | 18:43 | |
| pleia2 | this is for bare-trusty on hpcloud-b5 | 18:43 |
| clarkb | pleia2: thanks let me look at logs too | 18:43 |
| *** rfolco has joined #openstack-infra | 18:43 | |
| pleia2 | clarkb: yeah, all the hpcloud ones are foobar :\ | 18:44 |
| clarkb | pleia2: did you get ERROR: While executing gem ... (Errno::EACCES) ? | 18:44 |
| fungi | might be that hpcloud is full of hate right now. i've been having some trouble booting nodes on it too | 18:44 |
| *** cdent has quit IRC | 18:45 | |
| clarkb | I thought we had fixed that problem but maybe not in all cases | 18:45 |
| pleia2 | clarkb: all the hpcloud ones are foobar :\ | 18:45 |
| pleia2 | let's see exact error | 18:45 |
| pleia2 | 2015-02-24 18:38:45,565 INFO nodepool.image.build.hpcloud-b3.bare-trusty: ERROR: While executing gem ... (Errno::EACCES) | 18:45 |
| pleia2 | yep | 18:45 |
| clarkb | pleia2: that gem install error is what I see | 18:45 |
| *** carl_baldwin has joined #openstack-infra | 18:45 | |
| *** tnovacik_ has joined #openstack-infra | 18:45 | |
| notmyname | clarkb: so it seems like the 2 different invocations only differ by the config file used (I'm guessing a little). the first one uses the default (/etc/swift/test.conf) and the second one uses the one specified via environ | 18:46 |
| clarkb | pleia2: I would just skip hpcloud for the moment and get rax images done (since any broken images won't have latest requests anyways) | 18:46 |
| mordred | dtroyer: did you ever make a clouds.yaml devstack patch? | 18:47 |
| pleia2 | clarkb: right, ok | 18:47 |
| notmyname | clarkb: but i'm not sure how the /etc/swift/test.conf is set (or if the SWIFT_TEST_CONFIG_FILE is set to anything else for the first job) | 18:47 |
| *** vigneshvar has joined #openstack-infra | 18:48 | |
| *** krtaylor has quit IRC | 18:48 | |
| notmyname | clarkb: either way, I'd definitely support making the first one explicit as to the config file used. but I'm not sure how to separate it into 2 jobs. they only differ by the config file used, and I don't know the differences in them (ie what's in /etc/swift/test.conf) | 18:48 |
| *** cody-somerville has quit IRC | 18:48 | |
| clarkb | notmyname: basically you can take the env var setting that is done in the yaml file and put it into your tox.ini | 18:49 |
| clarkb | notmyname: so that when you run tox that makes tox just do the right thing | 18:49 |
| notmyname | clarkb: that's pretty dependent on the particular CI system you have, isn't it? eg I don't have /opt/stack/new/swift/... on my dev box | 18:50 |
| clarkb | pleia2: its odd that we don't have permissions issues in rax with that but do on hpcloud, I think we are just missing a sudo in front of the gem install line in our nodepool scripts | 18:50 |
| notmyname | clarkb: but anyway, how is /etc/swift/test.conf set? | 18:50 |
| *** tmcpeak has joined #openstack-infra | 18:50 | |
| *** sdake_ has joined #openstack-infra | 18:50 | |
| clarkb | notmyname: swift is configured by devstack in that job iirc | 18:51 |
| clarkb | notmyname: ya it may not be practical to have tox do it if config is very deployment specific | 18:51 |
| *** SumitNaiksatam has joined #openstack-infra | 18:51 | |
| *** marcusvrn1 has joined #openstack-infra | 18:52 | |
| clarkb | pleia2: I will write that change now and we can hopefully get it in while rax images are updated | 18:52 |
| *** marcusvrn has quit IRC | 18:52 | |
| *** zul has quit IRC | 18:52 | |
| pleia2 | clarkb: wfm | 18:52 |
| *** achanda has joined #openstack-infra | 18:52 | |
| *** zul has joined #openstack-infra | 18:52 | |
| notmyname | clarkb: that config file (ie the config section for functional tests) sets up the auth users and endpoint for an arbitrary swift cluster. I don't want to hard-code openstack jenkins slaves pathnames into the tox.ini. that seems wrong | 18:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Run restrict-memory element later in image build https://review.openstack.org/158413 | 18:53 |
| fungi | clarkb: ^ new stab at it | 18:53 |
| clarkb | pleia2: also I should check that we can install puppetlabs_spec_helper without that workaround now | 18:53 |
| clarkb | notmyname: ok | 18:53 |
| *** zul has quit IRC | 18:53 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:53 | |
| *** zul has joined #openstack-infra | 18:53 | |
| *** sdake__ has quit IRC | 18:54 | |
| *** tmcpeak has quit IRC | 18:54 | |
| *** kmartin has joined #openstack-infra | 18:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use sudo when gem installing during image build https://review.openstack.org/158822 | 18:55 |
| *** MarkAtwood has joined #openstack-infra | 18:55 | |
| clarkb | pleia2: ^ I confirmed that installing puppetlabs_spec_helper still fails on dependency resolution | 18:55 |
| clarkb | so sudo on the workaround should be what we need | 18:56 |
| * pleia2 nods | 18:56 | |
| *** tmcpeak has joined #openstack-infra | 18:56 | |
| *** tmcpeak has quit IRC | 18:56 | |
| *** adalbas has joined #openstack-infra | 18:57 | |
| *** changbl has quit IRC | 18:58 | |
| clarkb | almost meeting time | 18:59 |
| fungi | yep | 18:59 |
| jhesketh | Morning | 18:59 |
| clarkb | jhesketh: good morning | 18:59 |
| greghaynes | fungi: do youall have to support !ubuntu there? | 18:59 |
| fungi | we're driving in and out of dead zones, so i may be a little laggy to respond during the meeting | 18:59 |
| greghaynes | oh, ISWYDT | 18:59 |
| anteaya | fungi: k | 18:59 |
| *** jedimike has quit IRC | 18:59 | |
| clarkb | greghaynes: yes fedora and centos at least | 19:00 |
| *** krtaylor has joined #openstack-infra | 19:00 | |
| jeblair | meeting time in #openstack-meeting | 19:01 |
| BharatK | fungi, ran the job manually using your script | 19:02 |
| BharatK | fungi, https://etherpad.openstack.org/p/rax_console.txt | 19:02 |
| BharatK | fungi, VM went down for 569 ticks | 19:03 |
| fungi | BharatK: i'm still trying to get an hpcloud node. hpcloud is not terribly happy today | 19:03 |
| fungi | but souns like you were able to reproduce the failures we were seeing | 19:03 |
| BharatK | fungi, oh okay. | 19:03 |
| fungi | so that's a good sign | 19:03 |
| BharatK | fungi, yes able to reproduce. | 19:03 |
| *** emagana has quit IRC | 19:03 | |
| *** annegent_ has joined #openstack-infra | 19:03 | |
| *** emagana has joined #openstack-infra | 19:04 | |
| BharatK | fungi, after 569 ticks again accessible and ran some other test cases. | 19:04 |
| fungi | BharatK: yeah, that's roughly the behavior i was seeing as well | 19:04 |
| BharatK | fungi, yeah now the biggest challenge is finding the route cause. | 19:05 |
| BharatK | fungi, I think, I can run the same job again the same VM right | 19:05 |
| fungi | BharatK: well, and figuring out if you can reproduce elsewhere with those steps | 19:05 |
| BharatK | fungi, sudo I will do this in my local setup. | 19:06 |
| fungi | BharatK: you may be able to reproduce it by rerunning the same job on the same vm, but no guarantees there. we usually delete the vm after a job completes and use a freshly booted one every time | 19:06 |
| BharatK | fungi, yeah because I am planning do some modifications to the script and test it. | 19:06 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Added support for buildNeedsWorkspace entry in Build Flow Plugin https://review.openstack.org/150009 | 19:08 |
| *** ZZelle_ has joined #openstack-infra | 19:10 | |
| *** tmcpeak has joined #openstack-infra | 19:12 | |
| *** devvesa has quit IRC | 19:13 | |
| *** ghostpl_ has quit IRC | 19:13 | |
| *** tmcpeak has quit IRC | 19:13 | |
| *** jheroux has joined #openstack-infra | 19:13 | |
| *** FallenPegasus has joined #openstack-infra | 19:13 | |
| *** crc32 has joined #openstack-infra | 19:13 | |
| *** baoli has quit IRC | 19:14 | |
| *** Longgeek has joined #openstack-infra | 19:15 | |
| *** tonytan4ever has joined #openstack-infra | 19:17 | |
| *** jlibosva has quit IRC | 19:17 | |
| *** tonytan4ever has quit IRC | 19:20 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Default to dummy plugin info for test mode https://review.openstack.org/158826 | 19:20 |
| *** annegent_ has quit IRC | 19:23 | |
| *** ChuckC_ has joined #openstack-infra | 19:23 | |
| SpamapS | mordred: https://consul.io/ <-- thats the DNS-y thing that I was trying to recall while we were talking in Seattle btw. | 19:24 |
| *** hdd has quit IRC | 19:24 | |
| mordred | SpamapS: oh - yeah - I looked at that the other day | 19:24 |
| *** eharney has joined #openstack-infra | 19:24 | |
| mordred | SpamapS: but it's by hashicorp, so I just assume it will not work for me | 19:24 |
| *** teran has quit IRC | 19:25 | |
| greghaynes | Ive actually heard not bad things about it | 19:25 |
| SpamapS | True | 19:25 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for github commit status pending https://review.openstack.org/152818 | 19:25 |
| *** vigneshvar has quit IRC | 19:25 | |
| SpamapS | Well maybe they realized Packer and Vagrant were useless and they ought to make something some people can use. | 19:25 |
| *** ChuckC has quit IRC | 19:25 | |
| *** ChuckC has joined #openstack-infra | 19:26 | |
| mordred | SpamapS: I still haven't really understood why I want it - but I'm not violently against it like otherthings | 19:26 |
| greghaynes | It does seem to do a bit much though | 19:26 |
| *** annegent_ has joined #openstack-infra | 19:26 | |
| nibalizer | there was a good talk at fosdem about it | 19:27 |
| nibalizer | i would agree with greghaynes that does do a lot | 19:27 |
| clarkb | it seems a lot like dhcp in clouds | 19:27 |
| greghaynes | Theres also the thing of where if youre going to use it for DNS you kind of need to actually use it as your real resolver at which point youre not relying on something that has been production or battle tested like bind | 19:27 |
| clarkb | dhcp just works we should all use it | 19:27 |
| SpamapS | mordred: It's sort of like hiera but for all the things.. | 19:27 |
| *** ChuckC_ has quit IRC | 19:28 | |
| SpamapS | greghaynes: you can just use it for a subdomain of things you want to put in it. | 19:28 |
| * mordred agrees about sing actual dhcp and actual bind | 19:28 | |
| greghaynes | not to mention getting full resolution... or if you use a full resolver that talks to consol for some domains then whats the point of having the dns built into consul in the first place | 19:28 |
| SpamapS | greghaynes: I believe the original reasoning behind having DNS is that you don't have to write a consul->config file thing if you just can put 'http://some.host.in.consul.mydomain.foo/' in a static config file | 19:29 |
| SpamapS | also DNS is going to be faster than HTTP by 0.0001s for k/v lookups.. webscale baby | 19:30 |
| greghaynes | Sure, so for that case id just make a tool that pushes out zone transfers rather than implement a dns server | 19:30 |
| *** sdake_ has quit IRC | 19:31 | |
| SpamapS | greghaynes: except the whole zone has to be transferred to update things that consul updates when a client goes missing. | 19:31 |
| *** sdake_ has joined #openstack-infra | 19:31 | |
| *** prad has joined #openstack-infra | 19:31 | |
| *** carl_baldwin has quit IRC | 19:31 | |
| SpamapS | greghaynes: in this respect, ddns is the old way, and nobody seems to like ddns. | 19:31 |
| *** marcusvrn has joined #openstack-infra | 19:32 | |
| *** Sukhdev has joined #openstack-infra | 19:32 | |
| greghaynes | yeaaa, that is a lot of optimization | 19:32 |
| *** achanda has quit IRC | 19:33 | |
| anteaya | thingee uses packer since he can't get nodepool to do what he wants | 19:33 |
| SpamapS | greghaynes: you say that but this is very similar to Radware load balancers that existed 10 years ago and achieved global traffic re-routing within 3s of a problem being detected. | 19:33 |
| anteaya | which is work on a laptop | 19:33 |
| *** devlaps has quit IRC | 19:33 | |
| *** BharatK has quit IRC | 19:34 | |
| *** sdake__ has joined #openstack-infra | 19:34 | |
| anteaya | was going to bring that up before now, I just can never seem to figure out how to work it in the conversation | 19:34 |
| *** marcusvrn1 has quit IRC | 19:35 | |
| fungi | SpamapS: or for those of us who use the other thing, f5 3dns gtm | 19:35 |
| *** carl_baldwin has joined #openstack-infra | 19:35 | |
| *** dprince has quit IRC | 19:35 | |
| anteaya | so many things I learned at the mid-cycles and I can't figure out how to share it | 19:35 |
| clarkb | anteaya: if thingee just needs image builds I would suggest dib not nodepool | 19:36 |
| anteaya | he wants something to spin up vms for him | 19:36 |
| SpamapS | fungi: Yeah, 3DNS was a few years later, and similarly capable. The Radwares were nice because you could use snmp to poke stats at them pretty rapidly. The F5's thing had a complicated XML-RPC interface. | 19:36 |
| fungi | dib is certainly usable for building disk images | 19:36 |
| greghaynes | TIL | 19:36 |
| clarkb | anteaya: packer is an image builder not a VM manager | 19:37 |
| anteaya | ah | 19:37 |
| clarkb | anteaya: in some cases it does start a VM to build the image but thats orthogonal to its purpose | 19:37 |
| anteaya | I can ask him if he tried dib | 19:37 |
| *** sdake_ has quit IRC | 19:37 | |
| anteaya | thanks | 19:37 |
| anteaya | I'm finding a lot of projects making their own tools rather than using ours | 19:38 |
| afazekas | https://pypi.python.org/pypi/python-ceilometerclient The new ceilometer client is released today, and the icehouse jobs started require newer keystone-client | 19:38 |
| anteaya | since they don't fully know what all our tools do and they don't ask | 19:38 |
| viglesias | what are the advantages of dib over packer? | 19:39 |
| greghaynes | ^ We should codify that in the readme | 19:40 |
| SpamapS | viglesias: pretty rich composition. | 19:40 |
| greghaynes | that question comes up a lot | 19:40 |
| clarkb | the caching has been a huge win for us | 19:40 |
| SpamapS | viglesias: so you can break the image build bits into resuable bits. | 19:40 |
| viglesias | id also love to see a very simple quick start | 19:40 |
| viglesias | thats where packer had me, i understood it from the very basic case | 19:41 |
| *** achanda has joined #openstack-infra | 19:41 | |
| viglesias | extrapolating from there was also simple | 19:41 |
| greghaynes | ++ | 19:41 |
| mordred | agree | 19:41 |
| SpamapS | good point | 19:41 |
| viglesias | the README is more of a design doc | 19:41 |
| SpamapS | README is missing the basic getting started | 19:41 |
| mordred | someone go register dib.io | 19:41 |
| viglesias | dibs | 19:41 |
| greghaynes | That is on my to do list with the docs site being mad | 19:41 |
| viglesias | cool, just wanted to give you guys my 2c | 19:42 |
| greghaynes | Its super helpful :) | 19:42 |
| viglesias | for reference this is what i built out of packer: | 19:42 |
| viglesias | https://github.com/viglesiasce/cloud-images | 19:42 |
| SpamapS | mordred: Alex Graham from LeRoy, NY, owns it. | 19:42 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix bug in template job yaml expansion. https://review.openstack.org/150666 | 19:43 |
| viglesias | greghaynes: if you get that quick start, ill give it a go at the drop of a hat | 19:43 |
| *** hdd has joined #openstack-infra | 19:43 | |
| *** prad has quit IRC | 19:43 | |
| *** e0ne has quit IRC | 19:43 | |
| greghaynes | SpamapS: btw - this is one big downside of the dib-utils breakout | 19:43 |
| clarkb | viglesias: https://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh is roughly how we use it to make images that we run all of our tests on. note that openstack-repos and node-devstack caceh a lot of stuff so run it with lots of bandwidth the first time | 19:43 |
| greghaynes | SpamapS: We have to say "download this other repo, add it to your path..." | 19:44 |
| SpamapS | greghaynes: bash needs a cheeseshop. ;) | 19:44 |
| greghaynes | oh god | 19:44 |
| viglesias | stop it | 19:45 |
| viglesias | build it on top of pypi | 19:45 |
| mordred | SpamapS: GAH | 19:45 |
| SpamapS | godaddy's alt suggestions for bash.io are priceless | 19:45 |
| *** carl_baldwin has quit IRC | 19:45 | |
| viglesias | clarkb: so wrt your link, how do i know what an element is | 19:45 |
| SpamapS | you can have 'shindig.io' or 'bang.party' | 19:45 |
| *** eharney has quit IRC | 19:46 | |
| viglesias | or for that matter the most basic config to get a debian image lets say | 19:46 |
| *** HeOS has joined #openstack-infra | 19:46 | |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack-infra/system-config: Update mailman contact for openstack-fr mailing list https://review.openstack.org/158839 | 19:46 |
| clarkb | viglesias: thats beyond the scope of what we are doing there, the docs greghaynes says he will write should cover that | 19:46 |
| viglesias | kk | 19:46 |
| mordred | SpamapS: zomg. the website for dib should be bang.party | 19:46 |
| mordred | viglesias: one of the things I like about dib is taht the content is shell scripts, and not json or preseed files | 19:47 |
| mordred | but that's just me | 19:47 |
| SpamapS | I am cackling like an idiot about this | 19:47 |
| viglesias | mordred: word, i def use bash scripts with packer | 19:47 |
| viglesias | its common ground | 19:48 |
| viglesias | does dib support using kickstart or preseed files? | 19:48 |
| clarkb | viglesias: but to answer your question I think if you say `disk-image-create -o debian_image debian` you get what you want | 19:48 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Use sphinx extlinks to shorten links to Jenkins wiki https://review.openstack.org/158756 | 19:48 |
| clarkb | SpamapS: and greghaynes will tell me I am wrong now :) | 19:48 |
| greghaynes | you need an elemeent | 19:49 |
| SpamapS | Yep | 19:49 |
| SpamapS | debian is the element | 19:49 |
| viglesias | The First Element | 19:49 |
| SpamapS | vm too probably | 19:49 |
| greghaynes | oh, I see that ;) | 19:49 |
| pelix | zaro: looks like I could be playing whack-a-mole with https://review.openstack.org/158756, let me know if you think it's worth pursuing | 19:49 |
| SpamapS | the debian element is a little.. obtuse. It runs debootstrap every time.. so.. not the fastest thing | 19:49 |
| greghaynes | ubuntu vm ftw | 19:49 |
| SpamapS | though if you run it with --offline, it will reuse the last debootstrap | 19:50 |
| *** matrohon has joined #openstack-infra | 19:50 | |
| mordred | SpamapS: I really want to fix that | 19:50 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 19:50 | |
| *** diemt has joined #openstack-infra | 19:50 | |
| mordred | for some definition of fix | 19:50 |
| SpamapS | mordred: the trouble is detecting that it's ok to not build from scratch | 19:51 |
| SpamapS | ubuntu has the fact that they never publish more than one image a day | 19:51 |
| SpamapS | maybe a --moldy-cache flag | 19:52 |
| greghaynes | you just need to cache invalidate based on options + what you are debootstrapping? | 19:52 |
| greghaynes | oh, for new packages is what youre talking about | 19:52 |
| SpamapS | If you do stable Debian, you can just base it on the apt repo timestamps. | 19:52 |
| SpamapS | but unstable will update minute by minute. | 19:52 |
| SpamapS | testing too | 19:52 |
| mordred | sure- but the apt-get upgrade in subsequent images should make that fine | 19:52 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 19:52 | |
| mordred | s/images/elements | 19:53 |
| SpamapS | Simplest thing is probably to convince Debian to publish minimal images and base off those. | 19:53 |
| greghaynes | I like the --moldy-cache way | 19:53 |
| greghaynes | by default be fast, rebuild when someone asks for it | 19:53 |
| SpamapS | mordred: the apt get upgrade will be slower than just running debootstrap as the debootstrap gets older. | 19:53 |
| SpamapS | maybe we should keep the debootstrap for 3 days. | 19:54 |
| mordred | SpamapS: sure- but that can be an opt-in "--invalidate" or something | 19:54 |
| * mordred waves hands | 19:54 | |
| mordred | SpamapS: yeah - maybe 3 days | 19:54 |
| *** annegent_ has quit IRC | 19:54 | |
| SpamapS | or another idea.. | 19:54 |
| greghaynes | At my last job we made this tool, ran via cron every night to debootstrap a new container image | 19:54 |
| SpamapS | since we have a "separate cache building from image building" thing we need to do anyway.. | 19:54 |
| SpamapS | just help people setup cache building cron jobs. | 19:55 |
| SpamapS | and always run with --offline | 19:55 |
| SpamapS | greghaynes: GMTA | 19:55 |
| fungi | i believe there's been some traction on the debian-cloud ml to start generating base images for qemu/whatever | 19:55 |
| greghaynes | heh | 19:55 |
| fungi | zigo was pushing for that at one point anyway | 19:55 |
| SpamapS | fungi: yeah there are at least 3 separate groups | 19:56 |
| SpamapS | one wants EC2 only and gives everybody else the o_O look | 19:56 |
| fungi | and serve them alongside the cd images | 19:56 |
| SpamapS | one is google compute driven and is more accepting | 19:56 |
| SpamapS | and the other group is zigo | 19:56 |
| fungi | right | 19:56 |
| *** ddieterly has quit IRC | 19:56 | |
| mordred | JayF: the patchesyou have for cloud-init for the rax extensions | 19:56 |
| fungi | i feel bad i don't have more time to pitch in on those efforts | 19:56 |
| mordred | JayF: are those landed and just not released? or are they still pending something? | 19:56 |
| SpamapS | fungi: well they might be directly related to OpenStack activities... ;) | 19:57 |
| mordred | fungi, SpamapS thign is - I don't _WANT_ the base images from ubuntu or debian | 19:57 |
| SpamapS | fungi: so perhaps we should try and free you up to work on that. :) | 19:57 |
| mordred | I WANT the debootstrap method | 19:57 |
| *** ZZelle_ has quit IRC | 19:57 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for ironic bug 1425258 https://review.openstack.org/158844 | 19:57 |
| openstack | bug 1425258 in Ironic "test_list_baremetal_nodes race fails with a node not found 404" [Undecided,New] https://launchpad.net/bugs/1425258 | 19:57 |
| mordred | because the clodu images have garbage in them I don't want | 19:57 |
| SpamapS | mordred: if the base image from debian was debootstrap -> nothing else, you want that. | 19:57 |
| mordred | yes | 19:57 |
| mordred | if that's what it is | 19:57 |
| mordred | but it won't be | 19:57 |
| mordred | it'll have something else in it | 19:58 |
| mordred | like cloud-init | 19:58 |
| mordred | "to make it better" | 19:58 |
| SpamapS | "to make it actually do someting" might be how another perspective states that ;) | 19:58 |
| mordred | yes - but I don't want their version of it, and I don't want it not in a virtualenv | 19:58 |
| *** ddieterly has joined #openstack-infra | 19:58 | |
| mordred | honetly, grabbing the base docker image via docker, exporting it to a tar.gz and using that has step one | 19:58 |
| mordred | actually gets what I want | 19:58 |
| mordred | which is a VERY MINIMAL base image | 19:58 |
| SpamapS | that said, the group working on the google images also yelled at me for suggesting they include vim | 19:58 |
| SpamapS | so they MAY actually agree with you mordred. :) | 19:59 |
| clarkb | SpamapS: I get cranky when I don't have vim (how else do you debug things?) | 19:59 |
| mordred | I think the base image, as produced by a base dib element should not have it | 19:59 |
| SpamapS | clarkb: that's too much opinion for them. | 19:59 |
| mordred | but I think a "make this useful" element is a great idea | 19:59 |
| SpamapS | clarkb: some people get cranky when they DO have vim. | 19:59 |
| *** rm_work is now known as rm_work|away | 19:59 | |
| clarkb | SpamapS: they use ed? | 19:59 |
| SpamapS | or emacs | 19:59 |
| SpamapS | or nano | 19:59 |
| SpamapS | or DebianOMGwtfwhydowehavetohaveallthethings | 19:59 |
| *** sabeen3 has joined #openstack-infra | 20:00 | |
| SpamapS | Debian can't have nice things. | 20:00 |
| fungi | elvis | 20:00 |
| fungi | ;) | 20:00 |
| *** kgiusti1 has joined #openstack-infra | 20:00 | |
| mordred | this is why I think base images should not havethose things | 20:01 |
| clarkb | mordred: thats fair | 20:01 |
| clarkb | mordred: but reality is you will need a make this useful layer | 20:01 |
| *** teran has joined #openstack-infra | 20:01 | |
| SpamapS | mordred: right, if Debian had base + dib available.. thats a good story for users. | 20:01 |
| mordred | because, especially with dib composition, it's SUPER EASY to add | 20:01 |
| mordred | SpamapS: ++ | 20:01 |
| *** kgiusti has quit IRC | 20:01 | |
| fungi | but yeah, i expect any openstack-capable base images they publish officially to be geared toward at least being able to start in rackspace, hpcloud et cetera | 20:01 |
| clarkb | ok back to fixing the world | 20:01 |
| fungi | so possibly already opinionated in ways we wouldn't want | 20:02 |
| mordred | yah - I think there are two different things | 20:02 |
| *** sabeen1 has quit IRC | 20:02 | |
| mordred | I think there are base os images, and base cloud images | 20:02 |
| SpamapS | because 'disk-image-create -o debian-myway -p vim -p cloud-init debian vm' works | 20:02 |
| mordred | base cloud iamges should have enablement | 20:02 |
| jhesketh | jeblair, mordred: since we keep missing an opportunity to talk about a bower mirror during the infra meeting, if you have time to look at the latest version of https://review.openstack.org/#/c/154297/ that'd be great | 20:02 |
| jhesketh | just want to make sure it's on track | 20:02 |
| clarkb | fungi: canhazreview on https://review.openstack.org/#/c/158822/ that will unstick the devstack images on hpcloud | 20:02 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for ironic bug 1425258 https://review.openstack.org/158844 | 20:02 |
| openstack | bug 1425258 in Ironic "test_list_baremetal_nodes race fails with a node not found 404" [Undecided,New] https://launchpad.net/bugs/1425258 | 20:02 |
| clarkb | fungi: and ci will pass on them once we have the bare-* ndoes rebuilt | 20:02 |
| fungi | k | 20:02 |
| *** rm_work|away is now known as rm_work | 20:02 | |
| clarkb | pleia2: how are the bare-precise and bare-centos6 builds going? | 20:02 |
| *** sabeen3 has quit IRC | 20:02 | |
| *** sabeen2 has joined #openstack-infra | 20:03 | |
| *** teran has quit IRC | 20:03 | |
| pleia2 | clarkb: bare-trusty and bare-precise are all done | 20:03 |
| pleia2 | so is trusty-devstack | 20:03 |
| fungi | clarkb: need that force-submitted to gerrit? | 20:03 |
| fungi | clarkb: or will we have nodes capable of testing it successfully? | 20:03 |
| *** sandywalsh_ is now known as sandywalsh | 20:04 | |
| *** annegent_ has joined #openstack-infra | 20:04 | |
| clarkb | pleia2: trusty-devstack in rax? hpcloud? both? | 20:04 |
| pleia2 | only rax-dfw has finished bare-centos, it was being grumpy on the other two so I'm trying them again, also running precise-devstack (all three rax chugging along) | 20:04 |
| pleia2 | clarkb: everything only on rax | 20:04 |
| pleia2 | should devstacks work on hp? | 20:04 |
| clarkb | fungi: we should have nodes capable as soon as pleia2 is done with bare-centos6 and images cycle out | 20:04 |
| clarkb | pleia2: yes they should but lets hold off to get fungi's grub2 fix in | 20:05 |
| fungi | okay, i'll approve and let someone recheck it in as needed | 20:05 |
| clarkb | I am reviewing the grub2 fix now | 20:05 |
| pleia2 | alright | 20:05 |
| clarkb | fungi: is /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg going to work on centos? | 20:05 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceilometer 1.0.13 icehouse bug 1425262 https://review.openstack.org/158849 | 20:05 |
| openstack | bug 1425262 in python-ceilometerclient "1.0.13 release breaks stable/icehouse" [Undecided,New] https://launchpad.net/bugs/1425262 | 20:05 |
| * clarkb logs into a centos machine | 20:05 | |
| *** tim_o has joined #openstack-infra | 20:05 | |
| ianw | fungi: my first thought on bindep is that it seems to cover a lot of ground that puppet/ansible/etc have already figured out. i can see the appeal of it though | 20:05 |
| clarkb | fungi: looks like grub2-mkconfig exists on centos7 at least so this is a step in the right direction | 20:06 |
| fungi | ianw: i'm not terribly keen on running puppet during a jenkins job, nor does puppet have a great solution for "just download these packages but please don't install any of them yet" | 20:06 |
| clarkb | pleia2: 158413 if you want to review the grub2 fix as well | 20:07 |
| fungi | ianw: i should clarify, not keen on running puppet during _every_ jenkins job | 20:07 |
| fungi | ianw: because the idea is to be able to use one cross-platform manifest to tell nodepool what packages to cache and also to tell a job what packages to install before running tests | 20:07 |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack-infra/system-config: Update mailman contact for openstack-fr mailing list https://review.openstack.org/148949 | 20:08 |
| *** VijayTripathi has joined #openstack-infra | 20:08 | |
| *** tiswanso_ has quit IRC | 20:08 | |
| fungi | ianw: and to allow individual projects to easily list just the distro requirements they have for their tests (which is bindep's primary reason for existing actually) | 20:08 |
| *** _nadya_ has joined #openstack-infra | 20:08 | |
| clarkb | pleia2: I am +2 on 158413 and so is jeblair so feel free to approve if it looks ok to you (I will approve shortly otherwise) | 20:08 |
| *** koolhead17 has quit IRC | 20:09 | |
| fungi | ianw: the idea being you git clone nova, then run bindep and it will tell you what system packages you're missing (separate from what it will be doing with python requirements in a virtualenv) | 20:09 |
| zaro | pelix: yes, seems like it and not not real necessary. i would prefer a test to check for 404 error from those links. | 20:09 |
| pleia2 | clarkb: reviewing | 20:09 |
| fungi | ianw: no more trying to iteratively determine what dev packages you're missing by running tox -epy27 over and over | 20:10 |
| ianw | fungi: right, that's what i mean by i see the appeal of it | 20:10 |
| lifeless | jeblair: fungi: mordred: I'm glad to see bindep getting action; I'm happy for it to be in infra | 20:10 |
| fungi | ianw: so by also baking it into our node caching and job run-time, we further encourage projects to start making use of it, which in turn has beneficial side effects for devs too | 20:10 |
| lifeless | jeblair: fungi: mordred: if you are happy with me staying as core thats cool too | 20:11 |
| *** tmcpeak has joined #openstack-infra | 20:11 | |
| pelix | zaro: I'm guessing a unit test might be a little expensive there, so this something that can be run occassionally or part of the docs run to check for broken links? | 20:11 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 20:11 | |
| fungi | lifeless: i'm cool with it being an infra project with you as core on it, or having it remain in stackforge with you and/or other core reviewers as long as we're able to get fixes/releases for it | 20:12 |
| mordred | ++ | 20:12 |
| anteaya | lifeless: would you like to propose a patch to move it into infra, with it's own core group? | 20:12 |
| *** _nadya_ has quit IRC | 20:12 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:13 | |
| zaro | pelix: yeah, hoping there's a sphinx extension for that? | 20:13 |
| *** carl_baldwin has joined #openstack-infra | 20:13 | |
| pleia2 | fungi: just one quick comment inline on 158413 | 20:14 |
| fungi | yeah? | 20:14 |
| pleia2 | commented inline, the sed command changed so it adds a space in the mem= field now, intentional? | 20:15 |
| pleia2 | I don't know how finicky grub is about such things | 20:15 |
| clarkb | oh hrm let me see | 20:15 |
| zaro | pelix: otherwise i think just a shell script to check isn't expensive. probably would use curl? | 20:15 |
| pelix | zaro: well sphinx docs suggest that '-b linkcheck' should do the job | 20:16 |
| zaro | ohh cool | 20:16 |
| fungi | pleia2: replied | 20:16 |
| *** hdd has quit IRC | 20:16 | |
| pleia2 | fungi: gotcha, thanks :) | 20:16 |
| clarkb | is the quoting right there? | 20:17 |
| *** doude has quit IRC | 20:17 | |
| clarkb | this may be an actual problem | 20:17 |
| pleia2 | +3ed | 20:17 |
| *** otter768 has joined #openstack-infra | 20:17 | |
| pleia2 | doh | 20:17 |
| clarkb | oh wait no I see | 20:17 |
| clarkb | its correct | 20:17 |
| fungi | clarkb: the actual problem as seen in the logs was that the config files weren't present and so the if blocks were skipped | 20:17 |
| fungi | the new code is copied verbatim from restrict_memory.sh i believe | 20:17 |
| fungi | just with sudo trimmed out | 20:18 |
| clarkb | fungi: gotcha | 20:18 |
| clarkb | so dib and the other script got away from each other | 20:18 |
| fungi | but please double-check | 20:18 |
| lifeless | anteaya: nope | 20:18 |
| *** doude has joined #openstack-infra | 20:18 | |
| clarkb | fungi: yup you are right | 20:18 |
| fungi | i'm on finicky wireless from a car and about to pull over for a late lunch | 20:18 |
| anteaya | lifeless: oh okay | 20:18 |
| lifeless | anteaya: fungi is driving work on it, no need for me to frazzle my focus even further | 20:18 |
| clarkb | pleia2: see nodepool/scripts/restrict_memory.sh for the sibling script | 20:18 |
| lifeless | fungi: whatever works for you | 20:18 |
| clarkb | fungi: no worries | 20:19 |
| clarkb | cool once that gets in (assuming it passes the requests gauntlet) we should build hpcloud devstack-trusty image with dib then upload it to the hpcloud providers | 20:19 |
| fungi | lifeless: cool, i'm actually headed on a 10-day vacation with very little internet access, so may not get back to it until i get home again. if anybody wants to pick it up and run with it i'm working on getting stuff into a summarizable state | 20:19 |
| anteaya | fungi: do you want it to be an openstack-infra repo? | 20:20 |
| fungi | anteaya: i have no preference where that is concerned | 20:20 |
| morganfainberg | clarkb, fungi, something is awfully wrong with the pypi thing for python-keystoneclient-kerberos... i'm about to remove the project and try and re-add it. | 20:21 |
| morganfainberg | i'm not sure what else to do. | 20:21 |
| anteaya | fungi: okay I won't bother then if noone wants it | 20:21 |
| clarkb | the pypi thing? | 20:21 |
| morganfainberg | yeah | 20:21 |
| clarkb | morganfainberg: can you be more specific? | 20:21 |
| ianw | fungi: yeah, i don't want to be negative. just when you get into stuff like mysql/mariadb on various versions of fedora/rhel it would be nice(r) to leverage where people had figured it out before | 20:21 |
| morganfainberg | sorry | 20:21 |
| morganfainberg | uh | 20:21 |
| morganfainberg | publish to pypi is not working for it | 20:21 |
| morganfainberg | i've tried a few tags | 20:21 |
| clarkb | morganfainberg: gotcha, give me a minute to check on a couple things | 20:21 |
| *** otter768 has quit IRC | 20:21 | |
| morganfainberg | it hasn't worked, i've tried a few tags and other things on pypi side [ensuring permissions etc] | 20:22 |
| fungi | ianw: i agree. that part is fun but somewhat tractable. for example i looked at the puppet mysql module and it doesn't really do much to help there (for example doesn't do clients, dev libs, et cetera) | 20:22 |
| morganfainberg | now it is giving me 400s/500s when i try and reach it | 20:22 |
| pleia2 | ok, bare-precise, bare-trusty, devstack-precise, devstack-trusty and centos6-bare all done on the 3 rax regions, except devstack-precise on rax-ord which should hopefully be done soon | 20:22 |
| pleia2 | clarkb: what next? | 20:22 |
| fungi | okay, i'm disappearing for a few hours but will be back on from the hotel later | 20:23 |
| pleia2 | ah, there, the last rax-ord just finished | 20:23 |
| clarkb | pleia2: I think next step is getting grub fix in and the sudo gem install fix in (which should be doable with the new rax images) then we rebuild hpcloud images | 20:23 |
| *** marun has quit IRC | 20:23 | |
| clarkb | morganfainberg: was it ever registered on pypi? | 20:23 |
| clarkb | morganfainberg: I get does not exist as well | 20:23 |
| anteaya | fungi: k | 20:23 |
| morganfainberg | it is registered | 20:23 |
| pleia2 | clarkb: ok, how about the -dib ones while I wait? | 20:23 |
| morganfainberg | i can see some of the pages for it | 20:23 |
| morganfainberg | but not others | 20:23 |
| morganfainberg | and i'm an owner | 20:23 |
| clarkb | pleia2: those aren't actually used right now, only kept them because I thought I would be testing rax with dib much earlier | 20:23 |
| anteaya | pleia2: nice work | 20:23 |
| pleia2 | clarkb: ok, good | 20:23 |
| clarkb | pleia2: so you can ignore those for now (they will update on the daily update) | 20:24 |
| morganfainberg | clarkb, and openstackci is also an owner | 20:24 |
| pleia2 | anteaya: thanks | 20:24 |
| *** ghostpl_ has joined #openstack-infra | 20:24 | |
| clarkb | morganfainberg: it almost sounds like pypi's idea of that package is broken? maybe dstufft can help us | 20:24 |
| *** tmcpeak has quit IRC | 20:24 | |
| clarkb | dstufft: ^ python-keystoneclient-kerberos is in a funny state on pypi.python.org, any ideas? | 20:24 |
| morganfainberg | i can add you or dstufft as an owner if it would help | 20:24 |
| *** tmcpeak has joined #openstack-infra | 20:24 | |
| clarkb | morganfainberg: dstufft is pypi admin so shouldn't need that | 20:25 |
| dstufft | I can add myself as owner to anything ;) | 20:25 |
| dstufft | all your packages are belong to me | 20:25 |
| clarkb | ohnoes | 20:25 |
| anteaya | all the problems too sounds like | 20:25 |
| dstufft | yes :( | 20:25 |
| dstufft | morganfainberg: what version are you trying to upload | 20:25 |
| morganfainberg | dstufft, 0.1.2 was the one i just tagged via gerrit | 20:26 |
| dstufft | was that ever tagged before? | 20:26 |
| clarkb | dstufft: fwiw I get that package doesn't exist when I try to search for it | 20:26 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for ceilometer 1.0.13 icehouse bug 1425262 https://review.openstack.org/158849 | 20:26 |
| morganfainberg | dstufft, no, we had 0.1.0, 0.1.1, and now 0.1.2 | 20:26 |
| openstack | bug 1425262 in python-ceilometerclient "1.0.13 release breaks stable/icehouse" [Undecided,New] https://launchpad.net/bugs/1425262 | 20:26 |
| clarkb | not the version the entire name | 20:26 |
| dstufft | don't trust PyPI's web ui | 20:26 |
| clarkb | dstufft: :) ok | 20:26 |
| morganfainberg | we never tagged it before these versions | 20:26 |
| dstufft | https://warehouse.python.org/project/python-keystoneclient-kerberos/ | 20:26 |
| morganfainberg | it was only ever tagged afaik through openstack infra | 20:26 |
| dstufft | there are no releases registered | 20:27 |
| morganfainberg | right. | 20:27 |
| morganfainberg | something is really broken with it | 20:27 |
| dstufft | did you delete them? | 20:27 |
| clarkb | morganfainberg: dstufft let me dig up a log for the jobs that should publish the tag | 20:27 |
| morganfainberg | no. | 20:27 |
| *** dalgaaf has quit IRC | 20:27 | |
| morganfainberg | at least i'm not aware of deleting any | 20:27 |
| pelix | zaro: funnily enough it doesn't appear my changes that have broken links :p | 20:27 |
| morganfainberg | dstufft, nothing ever has published, but i didn't create this project in the normal way, someone else did | 20:27 |
| pelix | although there are a few | 20:27 |
| morganfainberg | i'm just trying to solve what the heck is going on | 20:28 |
| *** ghostpl_ has quit IRC | 20:28 | |
| dstufft | lemme sudo myself | 20:28 |
| *** ghostpl_ has joined #openstack-infra | 20:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Document Cross-Repo Dependencies https://review.openstack.org/158855 | 20:28 |
| morganfainberg | dstufft, i'd be happy to totally delete the project if it can be reregistered in the way i know works (from other registrations) | 20:28 |
| morganfainberg | dstufft, basically there is nothing important there now :) | 20:28 |
| AJaeger_ | jeblair: I documented CRD now based on your announcement. I only did minimal changes to your text. See change 158855 for details. | 20:29 |
| *** tmcpeak has quit IRC | 20:29 | |
| clarkb | morganfainberg: dstufft http://logs.openstack.org/a1/a1da205f26fa1d7694a94c80bd7d612e097a0f99/release/python-keystoneclient-kerberos-pypi-both-upload/8735cd4/console.html#_2015-02-24_20_17_46_114 | 20:29 |
| morganfainberg | clarkb, openstackci is an owner/maintainer | 20:29 |
| clarkb | morganfainberg: I don't see that because the package doesn't exist in pypi as far as I can tell | 20:30 |
| clarkb | morganfainberg: which is why I am confused | 20:30 |
| ianw | fungi: yeah ... maybe a model where somebody figures that stuff out once and extracts it to some sort of abstraction which can then be sourced. every project that has mysql dependencies having to figure it out over and over, or copy-paste and skew everywhere isn't appealing | 20:30 |
| morganfainberg | right and i see it :P | 20:30 |
| clarkb | OHHHH | 20:30 |
| clarkb | wait | 20:30 |
| morganfainberg | clarkb, oh god | 20:31 |
| morganfainberg | python-keystoneclient-kerberos | 20:31 |
| openstackgerrit | Merged openstack-infra/project-config: Run restrict-memory element later in image build https://review.openstack.org/158413 | 20:31 |
| clarkb | your package is keystoneclient-kerberos not python-keystoneclient-kerberos | 20:31 |
| clarkb | ya so you should fix that | 20:31 |
| openstackgerrit | Merged openstack-infra/project-config: Fix openstack-specs ACLs typo https://review.openstack.org/158666 | 20:31 |
| morganfainberg | yeah...... god | 20:31 |
| morganfainberg | clarkb, thanks... | 20:31 |
| morganfainberg | this has been driving me nuts | 20:31 |
| dstufft | I love things that I don't have to solve | 20:31 |
| *** unicell has quit IRC | 20:31 | |
| morganfainberg | the logs disappeared too quickly for me to see via zuul | 20:31 |
| morganfainberg | so made it hard to chase down | 20:31 |
| openstackgerrit | Merged openstack-infra/project-config: Use sudo when gem installing during image build https://review.openstack.org/158822 | 20:32 |
| dkranz | fungi: Was the idea to have a way to trigger a single experimental job on a project rather than all of them ever implemented? | 20:32 |
| clarkb | morganfainberg: fwiw if you do `git show-ref` it will show you the ref of your tag, then take that ref and go to http://git.openstack.org/ref[0:2]/ref for the logs | 20:32 |
| clarkb | morganfainberg: I wish we had a better way of publishing those things but ETOOMANYLOGS | 20:32 |
| *** garyh has quit IRC | 20:33 | |
| *** garyh has joined #openstack-infra | 20:34 | |
| clarkb | pleia2: looks like those changes merged, we will need the changes on nodepool.o.o before we continue. `sudo puppet agent --test` should manually update things if you don't want to wait for ansible | 20:34 |
| clarkb | pleia2: but no rush, I need lunch so I can wait :) | 20:35 |
| *** melwitt has joined #openstack-infra | 20:35 | |
| pleia2 | clarkb: yeah, maybe doing lunch first makes sense before I dive into this | 20:35 |
| openstackgerrit | Syed Ismail Faizan Barmawer proposed openstack-infra/project-config: Add ironic-lib to openstack https://review.openstack.org/157636 | 20:36 |
| morganfainberg | clarkb yeah no worries man. thanks for helping | 20:37 |
| morganfainberg | i'll get a fix proposed shortly to make this not sucky ;) | 20:37 |
| morganfainberg | and all that | 20:37 |
| morganfainberg | clarkb, really appreciate the help | 20:37 |
| clarkb | morganfainberg: no problem | 20:37 |
| *** teran has joined #openstack-infra | 20:38 | |
| pelix | zaro: should this be added to the documentation generation or do we want a separate test non-voting for this? | 20:39 |
| *** hdd has joined #openstack-infra | 20:39 | |
| AJaeger_ | pleia2: thanks for fixing the images! | 20:39 |
| *** dustins has quit IRC | 20:40 | |
| asselin | is devstack icehouse broken? | 20:40 |
| *** prad has joined #openstack-infra | 20:40 | |
| *** teran_ has joined #openstack-infra | 20:41 | |
| *** tiswanso has joined #openstack-infra | 20:44 | |
| *** teran has quit IRC | 20:44 | |
| anteaya | asselin: what do you see that prompts taht question? | 20:44 |
| asselin | my icehouse tests are failing...trying to see if it's just me | 20:45 |
| anteaya | can you show a paste? | 20:45 |
| mtreinish | asselin: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057699.html | 20:45 |
| JayF | mordred: not upstreamed yet :( | 20:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Document Cross-Repo Dependencies https://review.openstack.org/158855 | 20:46 |
| openstackgerrit | Merged openstack-infra/project-config: Bandit is a stackforge project https://review.openstack.org/156774 | 20:46 |
| asselin | anteaya, mtreinish here's my devstack output. I'll see if it's the same issue: http://paste.openstack.org/show/181421/ | 20:47 |
| mtreinish | asselin: yes | 20:48 |
| mtreinish | asselin: look in your swift logs you'll see the reqs mismatch on keystoneclient with ceiloclient | 20:48 |
| *** kgiusti1 has quit IRC | 20:48 | |
| mordred | JayF: k. cause I was going to get people to yell at people if they weren't taking your patches | 20:49 |
| *** marzif_ has joined #openstack-infra | 20:49 | |
| mtreinish | asselin: for example: http://logs.openstack.org/12/158012/3/check/check-tempest-dsvm-full-icehouse/09191e4/logs/screen-s-proxy.txt.gz | 20:49 |
| mordred | JayF: I've decided to break down and use your patched version of cloud-init running in a venv | 20:49 |
| *** ddieterly has quit IRC | 20:50 | |
| *** ddieterly has joined #openstack-infra | 20:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove wasteful tripleo workspace-cache cloning https://review.openstack.org/157490 | 20:51 |
| *** adalbas has quit IRC | 20:51 | |
| mestery | fungi clarkb: Any chance you can save the VM from this job (https://review.openstack.org/#/c/158076/) for me? I'm trying to debug a gate issue with ODL in the gate and live debug would be immensly helpful. | 20:52 |
| clarkb | meatery can you link to the running job you want held? | 20:53 |
| mestery | clarkb: Grabbing it ... | 20:53 |
| clarkb | heh mestery ^ | 20:53 |
| mestery | clarkb: https://jenkins05.openstack.org/job/check-tempest-dsvm-networking-odl/6/ | 20:53 |
| mestery | the -dvsm job there | 20:53 |
| mestery | meatery ... nice :) | 20:53 |
| anteaya | Building remotely on devstack-trusty-rax-dfw-989047 (devstack-trusty) | 20:54 |
| *** tiswanso has quit IRC | 20:54 | |
| mestery | I've been trying to debug an issue with iptables that's baffling me, some live debugging would likely let me solve it fairly quickly | 20:54 |
| clarkb | and since today is a nodepool howto day I ran sudo nodepool hold 989047 (that number suffix is the nodepool id) | 20:55 |
| clarkb | pleia2: ^ if you are interested | 20:55 |
| clarkb | mestery: where can I find an ssh pubkey? | 20:55 |
| clarkb | s/an/your/ | 20:55 |
| mestery | clarkb: Let me paste it, one sec | 20:55 |
| anteaya | mestery: for reference that number is on line 3 of https://jenkins05.openstack.org/job/check-tempest-dsvm-networking-odl/6/console | 20:55 |
| mestery | clarkb: http://paste.openstack.org/show/181437/ | 20:56 |
| mestery | anteaya: Thanks! :) | 20:56 |
| anteaya | sure | 20:56 |
| anteaya | happy digging | 20:56 |
| mestery | anteaya: :) thanks! | 20:57 |
| pleia2 | clarkb: ah nice, so someone can log in and poke around? | 20:57 |
| clarkb | mestery: jenkins@104.239.137.202 you may want to let the job finish firstso you aren't fighting it with your iptables changes | 20:57 |
| anteaya | mikal: thank you | 20:57 |
| mestery | clarkb: Ack, I'll keep an eye on it, thanks! | 20:57 |
| clarkb | pleia2: yup, that tells nodepool to not delete the node, then I typically just add the requestors ssh key to the jenkins authorized keys file | 20:57 |
| pleia2 | cool | 20:58 |
| clarkb | pleia2: then when they are done with it `sudo -H -u nodepool nodepool delete 989047` | 20:58 |
| *** baoli has joined #openstack-infra | 20:58 | |
| pleia2 | even cooler: I'm ordering beer in my grocery delivery (I lived in Pennsylvania before, this kind of thing would never happen) | 20:58 |
| * pleia2 nods | 20:58 | |
| clarkb | we try not to hold nodepool nodes a bunch and encourage local reproduction, but networking is a thing that is hard to reproduce locally since our clouds both do it in different and exciting ways | 20:59 |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack-infra/system-config: Removing inactive admins from root access https://review.openstack.org/158862 | 20:59 |
| mestery | clarkb: I only come to you when I've exhausted all other options ;) | 20:59 |
| * pleia2 nods | 20:59 | |
| clarkb | mestery: yup no wirries | 20:59 |
| clarkb | *worries | 20:59 |
| anteaya | mestery: we know, which is why you get a node held | 20:59 |
| mestery | Heh, that's like getting a gold star :) | 21:00 |
| AJaeger_ | anteaya, governance change for defcore repo has merged - feel free to review and approve https://review.openstack.org/#/c/156807 | 21:00 |
| anteaya | someone requested one held the other day, turns out he didn't know how to read the log files | 21:00 |
| mestery | :( | 21:00 |
| mestery | That's no fun | 21:00 |
| anteaya | mestery: well your version of gold and mine differ | 21:00 |
| mestery | lol | 21:00 |
| anteaya | mestery: we helped him learn how to reaad the log files | 21:00 |
| *** cmluciano has joined #openstack-infra | 21:02 | |
| *** hdd has quit IRC | 21:02 | |
| cmluciano | reed: Free for a PM? | 21:02 |
| anteaya | AJaeger_: thanks, done | 21:03 |
| AJaeger_ | thanks, anteaya ! | 21:03 |
| *** ihrachyshka has quit IRC | 21:03 | |
| *** tiswanso has joined #openstack-infra | 21:03 | |
| anteaya | sure | 21:03 |
| anteaya | thanks for keeping an eye on it | 21:03 |
| clarkb | pleia2: when I was in seattle amazon fresh would do it but you had to be home for the delivery so they could check your ID | 21:04 |
| *** annegent_ has quit IRC | 21:04 | |
| clarkb | pleia2: made it less practical when I wasn't working from home :) | 21:04 |
| clarkb | where it == deliver beer with groceries | 21:04 |
| *** kgiusti has joined #openstack-infra | 21:04 | |
| *** kgiusti has quit IRC | 21:04 | |
| *** hdd has joined #openstack-infra | 21:05 | |
| pleia2 | clarkb: yeah, they require the same here, but I do WFH | 21:05 |
| *** ihrachyshka has joined #openstack-infra | 21:06 | |
| *** teran_ has quit IRC | 21:08 | |
| *** teran has joined #openstack-infra | 21:08 | |
| *** ZZelle_ has joined #openstack-infra | 21:08 | |
| *** salv-orlando has quit IRC | 21:08 | |
| zaro | pelix: i prefer to add it to doc generation so it can be tested with tox -edocs ? | 21:08 |
| clarkb | pleia2: still waiting on puppet to run on nodepool.o.o. judging by http://puppetboard.openstack.org/node/nodepool.openstack.org it should be soon | 21:09 |
| pleia2 | ok | 21:09 |
| * clarkb pops out for a few more minutes since its not ready | 21:09 | |
| *** rfolco has quit IRC | 21:10 | |
| *** aysyd has quit IRC | 21:11 | |
| *** baoli has quit IRC | 21:11 | |
| *** andreykurilin_ has joined #openstack-infra | 21:12 | |
| *** baoli has joined #openstack-infra | 21:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Add a project to track DefCore reference data https://review.openstack.org/156807 | 21:12 |
| JayF | mordred: yell at them to move to github instead :(( | 21:13 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Check external documentation links are valid https://review.openstack.org/158871 | 21:13 |
| pelix | zaro: there we go ;) | 21:13 |
| anteaya | clarkb: when you return, this merged: https://review.openstack.org/#/c/158666/ so deprecating and hiding tc-chair gerrit group should be good to go | 21:13 |
| anteaya | and I don't think ttx is approving any patches in the cross-project meeting | 21:14 |
| *** changbl has joined #openstack-infra | 21:14 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Use sphinx extlinks to shorten links to Jenkins wiki https://review.openstack.org/158756 | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| SpamapS | mordred: Poking into shade.. one thing.. http://www.quickmeme.com/img/03/0347c3efdc17cc1959d089f60b8b2fc267d9093caa8e8cb483bf476b58e63e45.jpg | 21:16 |
| mordred | SpamapS: yeah. I hear that | 21:17 |
| mordred | SpamapS: rcarrillocruz has some patches up to start to add functional tests | 21:17 |
| *** andreykurilin_ has quit IRC | 21:17 | |
| mordred | to run against a devsatck | 21:17 |
| *** andreykurilin__ has joined #openstack-infra | 21:17 | |
| asselin | which pip modules is this in? from common import run_local | 21:17 |
| asselin | https://github.com/openstack-infra/project-config/blob/master/nodepool/scripts/cache_devstack.py#L23 | 21:17 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Make keystoneclient functional tests gating https://review.openstack.org/158872 | 21:18 |
| *** ParsectiX has quit IRC | 21:19 | |
| *** ParsectiX has joined #openstack-infra | 21:19 | |
| pelix | Is it possible for zuul to provide a link to the console output from a docs build run as well as a link to the generated docs? | 21:21 |
| *** sandywalsh has quit IRC | 21:22 | |
| *** andreykurilin_ has joined #openstack-infra | 21:22 | |
| *** andreykurilin__ has quit IRC | 21:22 | |
| AJaeger_ | pelix: yeah, that would be cool. | 21:23 |
| AJaeger_ | right now it's either or | 21:23 |
| clarkb | anteaya: noted, thanks | 21:23 |
| *** sandywalsh has joined #openstack-infra | 21:23 | |
| pelix | oh, well, here's hoping | 21:23 |
| clarkb | pelix: AJaeger_ should be as simple as s/docs-draft/logs/ in the url right? at least as the workaround for now | 21:23 |
| clarkb | zuul would probably take an option to link the default logs path and the specified job override too | 21:24 |
| AJaeger_ | clarkb: yeah, that is the "manual" workaround | 21:24 |
| *** Longgeek has quit IRC | 21:24 | |
| *** rlandy has quit IRC | 21:24 | |
| AJaeger_ | pelix: if you figure something out, it would be great ;) | 21:24 |
| mrodden | so has anyone else hit an "AttributeError: 'OrderedDict' object has no attribute 'extend'" while using jjb 1.1.0 ? | 21:24 |
| openstackgerrit | Michael Johnson proposed openstack-infra/git-review: Rebase to tracked remote/branch if defaultbranch=--track https://review.openstack.org/158877 | 21:24 |
| clarkb | pleia2: I double checked /etc/nodepool/scripts/prepare_node.sh on nodepool.o.o and it has the sudo gem install update in place | 21:25 |
| clarkb | pleia2: so I think you can try bare-trusty on hpcloud again | 21:25 |
| pleia2 | clarkb: alright, I'll try just one for now | 21:25 |
| *** jcoufal has joined #openstack-infra | 21:25 | |
| pc_m | Hi! I'm having problems with Jenkins on https://review.openstack.org/#/c/158389/3. It failed twice, on two different jobs, and when I look at the logs, I get a "File Not Found". Is there a know problem right now? | 21:26 |
| pleia2 | pc_m: I'm rebuilding images now for the fix | 21:26 |
| mrodden | whops nvm... i missed a dash in my yaml | 21:26 |
| clarkb | requests made a release that didn't accept rax's ssl cert which made log uploads to swift fail | 21:26 |
| notmyname | clarkb: yikes | 21:27 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: o.vo does not need old release testing https://review.openstack.org/158878 | 21:28 |
| pleia2 | yeah, fun times | 21:28 |
| pc_m | pleia2: Awesome! | 21:28 |
| clarkb | pleia2: are you also interested in dealing with the tc-chair group once images are happy? | 21:29 |
| * clarkb is selfishly teaching other people how to do things so that he doesn't have to do them as often >_> | 21:29 | |
| pleia2 | clarkb: sure, gerrit change? | 21:29 |
| *** MarkAtwood has quit IRC | 21:29 | |
| *** FallenPegasus is now known as MarkAtwood | 21:29 | |
| clarkb | pleia2: ya, can be done via ssh api or gui I think. I typically just gui it | 21:30 |
| *** garyh has quit IRC | 21:30 | |
| pleia2 | I'm happy to take the more mundane tasks off your plate so you can work on the hard stuff ;D | 21:30 |
| *** MarkAtwood has quit IRC | 21:30 | |
| clarkb | jeblair: sdague re https://review.openstack.org/#/c/156116/ can we decide if that is necessary? | 21:30 |
| *** garyh has joined #openstack-infra | 21:30 | |
| clarkb | alright I am really going to do the review.o.o db config update now /me logs into rax | 21:31 |
| *** garyh has quit IRC | 21:31 | |
| anteaya | clarkb: I'm glad because I am learning too | 21:31 |
| anteaya | clarkb: any time you want to teach me anything please do | 21:31 |
| *** garyh has joined #openstack-infra | 21:31 | |
| morganfainberg | clarkb, yay! i have a published -kerberos thing. thanks for the help | 21:31 |
| jeblair | clarkb: heh, if the initial version of the script had accidentally done that, i probably could deal with it. something about deliberately making a change to specifically make it incorrect rubs me the wrong way. | 21:32 |
| *** Sukhdev has quit IRC | 21:32 | |
| *** marzif_ has quit IRC | 21:32 | |
| sdague | jeblair: this is lining up with the apache format | 21:32 |
| jeblair | sdague: does apache use base 2 or 10? | 21:32 |
| sdague | don't know | 21:32 |
| clarkb | anteaya: kk | 21:32 |
| clarkb | morganfainberg: woot | 21:33 |
| anteaya | clarkb: thanks | 21:33 |
| pleia2 | clarkb, anteaya - so are we deleting tc-chair or unchecking "Make group visible to all registered users." ? | 21:33 |
| anteaya | I'm feeling frustrated that I'm not learning enough | 21:33 |
| pelix | zaro: I can fudge the check on the git clone url by turing it into an inline code block? | 21:33 |
| *** purp has joined #openstack-infra | 21:33 | |
| anteaya | clarkb: so that would help me | 21:33 |
| anteaya | pleia2: unchecking | 21:33 |
| clarkb | pleia2: we can't delete it so we need to uncheck the make group visible, I also like to remove all members and change ownership to administrators | 21:33 |
| jeblair | sdague: what do you want out of this? does it need to match apache exactly? | 21:33 |
| anteaya | pleia2: I'm told deleting isn't an option | 21:33 |
| clarkb | pleia2: and you can leave a comment in the comment box saying something like "group no longer in use" | 21:33 |
| sdague | but du -h also uses K, M, G, T | 21:33 |
| pleia2 | anteaya: yeah, I didn't see delete in the interface, but gerrit's interface is sneaky | 21:34 |
| anteaya | pleia2: 'tis | 21:34 |
| pleia2 | clarkb: on it | 21:34 |
| sdague | honestly, I find KiB to be really jarring to read (mental token parse is a lot longer) | 21:34 |
| sdague | and it's been this other way for a long time | 21:34 |
| jeblair | sdague: du has the interesting property that it doesn't actually specify the unit | 21:35 |
| jeblair | sdague: (eg, 'K' instead of 'KiB' or 'KB") | 21:35 |
| pleia2 | clarkb, anteaya: done | 21:35 |
| anteaya | pleia2: thank you | 21:36 |
| *** AJaeger_ has quit IRC | 21:36 | |
| sdague | jeblair: right | 21:36 |
| sdague | I want K | 21:36 |
| clarkb | I just updated the review.o.o timeout config via the rax dashboard | 21:36 |
| sdague | because your brain processes the first and last token of a word and skips the middle in first glance | 21:36 |
| clarkb | used the same config applied to review-dev which is called 'default-timeout' | 21:36 |
| sdague | so the last character being significant helps, at least me | 21:36 |
| clarkb | and review.o.o appeas to still work so \o/ | 21:37 |
| clarkb | pleia2: thanks a bunch | 21:37 |
| anteaya | ttx your tech-committee-chair group is up and the acl points to it: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/governance.config the tc-chair gerrit group is disabled | 21:37 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Check external documentation links are valid https://review.openstack.org/158871 | 21:37 |
| *** sdake has quit IRC | 21:37 | |
| *** sabeen3 has joined #openstack-infra | 21:37 | |
| clarkb | pleia2: is that hpcloud bare-trusty job beyond the gem install yet? | 21:38 |
| jeblair | sdague: i could go with 'K' under the theory it's too vague to matter :) | 21:38 |
| ttx | anteaya: ack | 21:39 |
| pelix | zaro: never mind just spotted your review comment | 21:39 |
| sdague | yeh, I don't want the B or the iB there, just the K :) | 21:39 |
| sdague | so we can interpret it however :) | 21:39 |
| pleia2 | clarkb: it's doing puppet things, I forget where that is in the install | 21:39 |
| jeblair | sdague: done! | 21:39 |
| *** sabeen2 has quit IRC | 21:39 | |
| crinkle | clarkb: Hunner just released new puppetlabs_spec_helper | 21:39 |
| clarkb | jhesketh: back yet? I never saw a list of IPs to check on the haproxy servers, let me know if you have those | 21:39 |
| openstackgerrit | Merged openstack-infra/project-config: Fix swift func tests to run using both auth versions https://review.openstack.org/158671 | 21:39 |
| crinkle | which appears to be not broken | 21:39 |
| clarkb | crinkle: Hunner thank you | 21:39 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Use sphinx extlinks to shorten links to Jenkins wiki https://review.openstack.org/158756 | 21:40 |
| clarkb | pleia2: I think puppet things are after so gem install probably worked | 21:40 |
| *** tnovacik_ has quit IRC | 21:40 | |
| *** marun has joined #openstack-infra | 21:40 | |
| Hunner | clarkb: Sorry it took so long. | 21:40 |
| jeblair | i just sent the infra-cloud email | 21:40 |
| anteaya | I see it | 21:41 |
| *** matrohon has quit IRC | 21:41 | |
| *** ParsectiX has quit IRC | 21:41 | |
| *** ParsectiX has joined #openstack-infra | 21:42 | |
| clarkb | Hunner: crinkle it gets further in my docker container but breaks on ERROR: Failed to build gem native extension. does that mean i am missing dev headers for a compile? | 21:42 |
| clarkb | (would not suprise me if that is the case since its a docker image) | 21:42 |
| *** dustins has joined #openstack-infra | 21:42 | |
| *** dustins has quit IRC | 21:42 | |
| Hunner | clarkb: psh is not a native gem, so something else? | 21:43 |
| Hunner | clarkb: I haven't read scrollback. What gem are you installing? | 21:43 |
| clarkb | Hunner: puppetlabs_spec_helper | 21:43 |
| crinkle | clarkb: you might need to install ruby-dev | 21:43 |
| clarkb | http://paste.openstack.org/show/181457/ | 21:44 |
| clarkb | Hunner: crinkle ^ | 21:44 |
| clarkb | crinkle: will try that | 21:44 |
| Hunner | clarkb: That's probably json | 21:44 |
| Hunner | not json-ruby... | 21:44 |
| clarkb | crinkle: that was it | 21:44 |
| crinkle | i think the mkmf thing is usually ruby-dev | 21:44 |
| crinkle | clarkb: :) | 21:45 |
| Hunner | Ah yep, json | 21:45 |
| Hunner | via nibalizer's metadata-json-lint dependency that is new as of 0.9.0 | 21:45 |
| clarkb | nibalizer: ! :P | 21:45 |
| pleia2 | clarkb: it failed differently this time, but looking at the screen buffer there's lots of red puppetlabs_spec_helper stuff | 21:46 |
| clarkb | new fails are always the best | 21:46 |
| *** unicell has joined #openstack-infra | 21:47 | |
| *** sdake has joined #openstack-infra | 21:47 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-zuul: Switch zuul paramiko dependency to pip provider https://review.openstack.org/158882 | 21:47 |
| clarkb | pleia2: maybe new release doesn't like use for some reason? | 21:47 |
| crinkle | logs we can look at? | 21:47 |
| asselin | clarkb, nibalizer ^^ there's gotta be a better way to manage dependencies in puppet, no? | 21:48 |
| *** sdake has quit IRC | 21:48 | |
| pleia2 | http://paste.openstack.org/show/181458/ | 21:48 |
| openstackgerrit | Merged openstack-infra/project-config: zuul-swift-log index size in units https://review.openstack.org/156116 | 21:48 |
| clarkb | asselin: are you on precise for nodepool? | 21:49 |
| crinkle | pleia2: that's the error clarkb was running into too | 21:49 |
| asselin | clarkb, yes | 21:49 |
| * pleia2 nods | 21:49 | |
| clarkb | asselin: we are trusty now which is 1.10.1 so we haven't noticed | 21:49 |
| clarkb | crinkle: pleia2 oh fun | 21:49 |
| asselin | clarkb, I see | 21:49 |
| asselin | clarkb, also, you don't install the two on the same box | 21:50 |
| clarkb | asselin: I am not sure there is a good answer there, there are several options but I don't think any are good :) | 21:50 |
| clarkb | asselin: correct we have zuul and nodepool et al separated | 21:50 |
| *** mrmartin has quit IRC | 21:50 | |
| clarkb | asselin: the best thing may be to document that older paramiko works fine just without ecdsa support | 21:50 |
| clarkb | asselin: then don't hard require 1.9.x | 21:50 |
| *** marun has quit IRC | 21:51 | |
| nibalizer | what i broke it? | 21:52 |
| clarkb | nibalizer: yes | 21:52 |
| asselin | clarkb, the real issue is nodepool has errors when it doesn't have the right version of paramiko. http://paste.openstack.org/show/181464/ | 21:52 |
| clarkb | apparently the json dep requires ruby-dev be installed so that puppetlabs_spec_helper installs | 21:52 |
| *** sdake has joined #openstack-infra | 21:53 | |
| nibalizer | sounds fixed now tho | 21:53 |
| clarkb | asselin: oh that changes things, so we do require newer paramiko for the pty stuff on fedora to make sudo work | 21:53 |
| clarkb | nibalizer: no its not fixed | 21:53 |
| nibalizer | < clarkb> crinkle: that was it ? | 21:54 |
| clarkb | nibalizer: I was testing in my docker container | 21:54 |
| clarkb | nibalizer: I am not sure I relly want to install ruby-dev on all nodes so that puppetlabs_spec_helper will install | 21:54 |
| clarkb | but maybe we need to | 21:54 |
| pleia2 | what's different about the rax systems that makes them not explode? | 21:55 |
| mordred | clarkb: wow, really? | 21:55 |
| Hunner | clarkb: I'll just 0.9.1 and remove the hard-dependency | 21:56 |
| crinkle | clarkb: Hunner's going to revert the metadata-json-lint to fix it | 21:56 |
| crinkle | jinx | 21:56 |
| Hunner | :) | 21:56 |
| nibalizer | boo | 21:56 |
| *** sdake_ has joined #openstack-infra | 21:56 | |
| nibalizer | so why does metadata-json-lint need ruby-dev? | 21:56 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set custom cpu_model for live_migrate https://review.openstack.org/141530 | 21:56 |
| Hunner | nibalizer: I know, but requiring dev toolchain is boo too | 21:56 |
| Hunner | nibalizer: Because it depends on json | 21:56 |
| Hunner | And json is native | 21:57 |
| nibalizer | how does puppet not depend on json | 21:57 |
| Hunner | pson ;) | 21:57 |
| Hunner | (puppet vendored json) | 21:57 |
| Hunner | Well, at least it used to | 21:57 |
| nibalizer | so no ruby library can use json without the user installing ruby-dev? | 21:57 |
| clarkb | Hunner thank you for the help | 21:57 |
| Hunner | There is json-ruby, but it's lame | 21:57 |
| Hunner | nibalizer: Basically | 21:57 |
| clarkb | pleia2: different base images, likely rax leaks the deps in for us | 21:57 |
| pleia2 | clarkb: ah, I suppose this might be an issue for the -dib images too then | 21:58 |
| openstackgerrit | Michael Johnson proposed openstack-infra/git-review: Rebase to tracked remote/branch if defaultbranch=--track https://review.openstack.org/158877 | 21:58 |
| *** sdake_ has quit IRC | 21:58 | |
| nibalizer | wow okay thats super dumb | 21:58 |
| clarkb | ya but we dont install this package there so hopefully we avoid it | 21:58 |
| nibalizer | ill just rewrite metdata-json-lint in bash | 21:59 |
| nibalizer | require jq | 21:59 |
| pleia2 | hehe | 21:59 |
| * nibalizer shuffles off to be angry in the corner | 21:59 | |
| *** mbacchi has quit IRC | 21:59 | |
| clarkb | pleia2: at this point probably easiest to just wait for hunner to fix upstream then try building again | 21:59 |
| clarkb | pleia2: do you want to give the dib images a go while we wait? | 22:00 |
| *** sdake__ has quit IRC | 22:00 | |
| pleia2 | clarkb: agreed, sure re: dib | 22:00 |
| clarkb | pleia2: start with `sudo -H -u nodepool nodepool image-build devstack-trust` that will just do the dib stuff to build a new devstack-trusty | 22:00 |
| pleia2 | then start uploading it to all the clouds | 22:01 |
| clarkb | then when that is done run `sudo -H -u nodepool nodepool image-upload hpcloud-b1` and so on | 22:01 |
| clarkb | yup | 22:01 |
| *** shashankhegde has joined #openstack-infra | 22:01 | |
| *** dustins has joined #openstack-infra | 22:01 | |
| clarkb | also my 'y' key doesn't like me | 22:01 |
| pleia2 | also need -dib | 22:01 |
| pleia2 | right? | 22:01 |
| clarkb | pleia2: nope | 22:01 |
| pleia2 | oh ok | 22:01 |
| clarkb | pleia2: we have two sets of dib images, the -dib suffix images which are basically the dib guinea pig labeled images | 22:01 |
| *** Sukhdev has joined #openstack-infra | 22:02 | |
| clarkb | pleia2: I should probably clean that up now | 22:02 |
| *** mwagner_lap has quit IRC | 22:02 | |
| pleia2 | I see | 22:02 |
| pleia2 | clarkb: then we build devstack-precise, how about devstack-centos7-dib ? | 22:04 |
| *** vilobhmm has quit IRC | 22:04 | |
| *** carl_baldwin has quit IRC | 22:04 | |
| *** vilobhmm1 has joined #openstack-infra | 22:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add ansible role for running puppet https://review.openstack.org/158173 | 22:05 |
| clarkb | pleia2: I think devstack-centos7-dib is in a similar place to the other -dib images so can be ignored for now | 22:05 |
| mordred | clarkb: ok. if I don't TOTALLY suck that one should kinda work | 22:05 |
| openstackgerrit | Merged stackforge/gertty: Add detailed examples and dashboards a la gerrit https://review.openstack.org/142108 | 22:07 |
| *** tmcpeak has joined #openstack-infra | 22:07 | |
| clarkb | hrm if I have a label in nodepool's yaml do I have to provide at least one provider under it? | 22:07 |
| * clarkb reads some code | 22:08 | |
| *** dangers_away is now known as dangers | 22:08 | |
| *** gyee has quit IRC | 22:09 | |
| *** Hefeweizen has quit IRC | 22:10 | |
| *** pelix has quit IRC | 22:10 | |
| SpamapS | hrm.. getting used to gertty.. long lines don't wrap in diff view. Thats weird. | 22:10 |
| clarkb | SpamapS: ya I usually just 'c' and git diff, its probably the only major problem with gertty that I have | 22:11 |
| clarkb | and worked around easily enough | 22:11 |
| jeblair | SpamapS: yeah, i think that might be a one-liner | 22:11 |
| *** bcourt has quit IRC | 22:12 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Cleanup devstack-(trusty|precise)-dib images https://review.openstack.org/158891 | 22:14 |
| clarkb | pleia2: ^ something like that would clean it up until needed again for rackspace | 22:14 |
| *** tmcpeak has quit IRC | 22:15 | |
| pleia2 | \o/ | 22:15 |
| *** rm_work is now known as rm_work|away | 22:16 | |
| *** radez is now known as radez_g0n3 | 22:16 | |
| *** gyee has joined #openstack-infra | 22:17 | |
| *** stevemar has quit IRC | 22:17 | |
| *** changbl has quit IRC | 22:17 | |
| *** marun has joined #openstack-infra | 22:17 | |
| *** otter768 has joined #openstack-infra | 22:18 | |
| *** gyee has quit IRC | 22:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Create Influxdb-Grafana Fuel plugin project in Stackforge https://review.openstack.org/156245 | 22:20 |
| *** carl_baldwin has joined #openstack-infra | 22:21 | |
| pleia2 | clarkb: I see, "diskimage:" being defined refers to a dib-built image, and we've migrated to using the non-dib names for those at the bottom of the file | 22:22 |
| clarkb | pleia2: correct | 22:22 |
| *** otter768 has quit IRC | 22:22 | |
| clarkb | pleia2: the reason I added the -dib stuff was so that we could continue using snapshot builds while I tested the dib builds worked with a different label | 22:22 |
| Hunner | clarkb: Try again? 0.9.1 is out | 22:23 |
| clarkb | but now that hpcloud is dib'd there and rax isn't ready yet we don't need to build those images every day and keep 1 node ready | 22:23 |
| clarkb | pleia2: ^ | 22:23 |
| pleia2 | got it | 22:23 |
| pleia2 | clarkb: do you want to try in docker first? | 22:24 |
| clarkb | pleia2: ya actually that is probably quicky | 22:24 |
| clarkb | s/quicky/quicker/ I should just stop typing today | 22:24 |
| jeblair | clarkb, jhesketh, mordred: i just ran this command: diff <(git log -p -1 changes/32/119532/9) <(git log -p -1 changes/32/119532/14) | 22:24 |
| mordred | clarkb: speaking of dib-ing rax - you wanna review some shade? | 22:25 |
| mordred | jeblair: dude. that's a command | 22:25 |
| clarkb | mordred: I have been reviewing some shade | 22:25 |
| mordred | clarkb: yay! | 22:25 |
| jeblair | clarkb, jhesketh, mordred: to interdiff those two patchsets (separated by a rebase), taking advantage of the fact that i have all the refs locally thanks to gertty | 22:25 |
| clarkb | mordred: not today but in general | 22:25 |
| *** gyee has joined #openstack-infra | 22:26 | |
| mordred | clarkb: yes - they have been good reviews | 22:26 |
| clarkb | pleia2: Hunner looks like puppetlabs_spec_helper installs now | 22:27 |
| clarkb | pleia2: so I say give the image build another go | 22:28 |
| *** diemt has quit IRC | 22:28 | |
| * pleia2 presses the big red button | 22:28 | |
| Hunner | clarkb: As long as you don't need to validate metadata.json :) | 22:28 |
| clarkb | Hunner: I have no idea :) | 22:29 |
| *** cody-somerville has joined #openstack-infra | 22:29 | |
| mordred | we could always just write a python script to validate metadata.json ... | 22:30 |
| Hunner | Right. You're just the messenger | 22:30 |
| Hunner | mordred: I mean, using the rake task that puppetlabs_spec_helper ships. | 22:30 |
| clarkb | nibalizer: ^ you probably know | 22:30 |
| *** tiswanso has quit IRC | 22:31 | |
| Hunner | Hmm. A ruby library that requires a python library... | 22:31 |
| mordred | Hunner: oh, I was just saying words, I haven't been following this at all | 22:31 |
| *** erlon has quit IRC | 22:31 | |
| *** ihrachyshka has quit IRC | 22:31 | |
| Hunner | mordred: Which is how many flashes of insight can occur. Even from a literal rubber duck | 22:31 |
| nibalizer | clarkb: we dont really need to validate metadata.json | 22:32 |
| jeblair | mordred: to be fair, you did write an ansible role to call puppet. :) | 22:32 |
| nibalizer | i think the solution is to switch out json for json_pure which doesn't need ruby-dev | 22:32 |
| Hunner | Anisble calling puppet is like the best use ever for ansible :P | 22:32 |
| mordred | jeblair: this is true | 22:32 |
| nibalizer | clarkb: we'll want to validate metadata.json real soon now | 22:32 |
| mordred | jeblair: and a puppet module to install and configure ansible | 22:32 |
| pleia2 | clarkb: so dib image, I want: sudo -H -u nodepool nodepool image-upload hpcloud-b1 devstack-trusty | 22:33 |
| mordred | Hunner: then you should enjoy this: https://gitorious.org/mordred-temp/ansible-puppet | 22:33 |
| *** salv-orlando has joined #openstack-infra | 22:33 | |
| clarkb | pleia2: yup after the image-build command is done and successful | 22:33 |
| Hunner | nibalizer: I wonder... what if you add a dependency on json_pure, then try to install metadata-json-lint as-is? | 22:33 |
| *** gyee has quit IRC | 22:34 | |
| openstackgerrit | John L. Villalovos proposed openstack/requirements: Avoid httpretty 0.8.7 as it can break unittests https://review.openstack.org/158897 | 22:34 |
| nibalizer | Hunner: what do you mean? | 22:34 |
| nibalizer | change only the dep? | 22:34 |
| nibalizer | that was my plan... | 22:34 |
| Hunner | nibalizer: I mean, without changing the dep. Does it complain or not? | 22:34 |
| Hunner | (like, does json_pure satisfy the dep on json) | 22:35 |
| *** jcoufal has quit IRC | 22:35 | |
| *** dkranz has quit IRC | 22:35 | |
| * nibalizer tries | 22:35 | |
| mordred | yay! this passes checks now: https://review.openstack.org/#/c/158173/ anteaya, clarkb, jeblair, sdague ... if we land that, I can submit the next patch to it once the new repo is there (and I can sheperd the project creation) | 22:35 |
| fungi | asselin: if nobody's answered you, it's being imported from that same directory as the script you linked | 22:35 |
| asselin | fungi, no one answered....so thank you. I'll have to look again now | 22:36 |
| pleia2 | clarkb: hrm http://paste.openstack.org/show/181489/ | 22:37 |
| *** openstackgerrit has quit IRC | 22:37 | |
| *** unicell has quit IRC | 22:37 | |
| *** openstackgerrit has joined #openstack-infra | 22:37 | |
| *** roaet has joined #openstack-infra | 22:37 | |
| *** sabeen1 has joined #openstack-infra | 22:38 | |
| clarkb | pleia2: it seems to have left off the .qcow2 extension for some reason | 22:39 |
| *** garyh has quit IRC | 22:39 | |
| pleia2 | indeed | 22:39 |
| pleia2 | /opt/nodepool_dib/devstack-trusty-1424815332.qcow2 does exist | 22:39 |
| clarkb | yup | 22:39 |
| *** sabeen3 has quit IRC | 22:40 | |
| *** bhuvan has quit IRC | 22:40 | |
| jlvillal | clarkb: fungi: Figured out that httpretty 0.8.7 was cause of issue for our failing gate jobs. https://review.openstack.org/158897 | 22:41 |
| *** baoli has quit IRC | 22:41 | |
| roaet | Howdy. Any hints on how I can debug? https://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron-src-python-neutronclient-icehouse/49/console I ran recheck once and I'd rather not thrash it. | 22:41 |
| *** baoli has joined #openstack-infra | 22:41 | |
| roaet | I checked LP and didn't get anything. Error is ERROR: the main setup script run by this job failed - exit code: 1 ... and the only thing on LP mentions pylint. | 22:42 |
| jeblair | roaet: the detailed logs are here: http://logs.openstack.org/18/148318/16/check/gate-tempest-dsvm-neutron-src-python-neutronclient-icehouse/9321066// | 22:42 |
| jeblair | roaet: that's the link that should be in gerrit | 22:43 |
| roaet | ah.. odd how I went to jenkins | 22:43 |
| *** dustins has quit IRC | 22:43 | |
| *** dustins has joined #openstack-infra | 22:43 | |
| clarkb | pleia2: I think I know why this is broken, give me a couple minutes to confirm | 22:44 |
| roaet | Ahhh glance. | 22:44 |
| pleia2 | clarkb: thanks | 22:44 |
| pleia2 | tsk, they didn't have the 6-pack of beer I wanted, case it is! | 22:45 |
| roaet | jeblair: is the right thing to do at this point: recheck BUG#? | 22:45 |
| roaet | i think I found the bug that is doing it | 22:45 |
| pleia2 | no need to include the bug number these days | 22:46 |
| roaet | oh. ok. | 22:46 |
| jeblair | roaet: if there isn't already an elastic-recheck signature for it, you might consider writing one | 22:46 |
| pleia2 | (I didn't confirm recheck is the right thing here, just a comment on syntax) | 22:46 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Append file type to filename on image upload https://review.openstack.org/158903 | 22:47 |
| *** nelsnelson has quit IRC | 22:47 | |
| roaet | jeblair: I sadly have no idea what you're talking about :( | 22:47 |
| clarkb | pleia2: ^ thats the fix if you look at I45732195754e97d287f3c4c3906a48e8d6f13ed7 you can see in the diff that we used to do that update to the filename var but now we don't | 22:47 |
| clarkb | mordred: jeblair ^ review on 158903 would be great so we can finish these image updates | 22:48 |
| clarkb | that'll fail pep8 and I am ashamed | 22:48 |
| clarkb | fixing | 22:49 |
| pleia2 | roaet: we now have a tool that helps classify bugs, it helps when developers contribute to the bug database for it: http://docs.openstack.org/infra/elastic-recheck/readme.html | 22:49 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Append file type to filename on image upload https://review.openstack.org/158903 | 22:49 |
| roaet | pleia2: checking it out | 22:49 |
| roaet | pleia2: I guess the docs are pretty new :/ | 22:51 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Utilize dogpile.cache for caching https://review.openstack.org/158906 | 22:52 |
| *** jheroux has quit IRC | 22:53 | |
| pleia2 | roaet: it's been around for a bit, just poorly linked I think, we now talk about it in our dev guide http://docs.openstack.org/infra/manual/developers.html#automated-testing | 22:53 |
| fungi | clarkb: pleia2: i managed to hold a new hpcloud centos7 vm and it is showing 8gb ram now, so looks like the mem limit worked | 22:54 |
| pleia2 | fungi: yay | 22:54 |
| *** achanda has quit IRC | 22:54 | |
| *** dustins has quit IRC | 22:55 | |
| arosen | clarkb: i hate to bug you again but my nodepool instances aren't going to used when jenkins picks them up. It looks like nodepool <---> jenkins communication is working though over gearman | 22:56 |
| nibalizer | arst | 22:56 |
| arosen | tcp 0 0 10.0.0.109:43178 10.0.0.91:4730 ESTABLISHED | 22:56 |
| arosen | any ideas what could be causing this? I just upgrade jenkins to 1.599 | 22:56 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Utilize dogpile.cache for caching https://review.openstack.org/158906 | 22:56 |
| arosen | I thought my issue last night was because I had upgraded jenkins (though it turned out i was confused about the binding order of zmq which you explained to me) | 22:57 |
| jhesketh | clarkb: re those IP's I got distracted fixing requests stuff on turbo-hipster yesterday so I don't have them... Not sure if they are readily available either because I haven't logged the IP's anywhere so I'll have to wait for it to happen again | 22:57 |
| openstackgerrit | Michael Johnson proposed openstack-infra/git-review: Rebase to tracked remote/branch if defaultbranch=--track https://review.openstack.org/158877 | 22:57 |
| *** mriedem has quit IRC | 22:57 | |
| clarkb | arosen: nodepool to jenkins is the zmq event publisher | 22:57 |
| clarkb | arosen: so double check its still binding on that port and that nodepool is connected to it | 22:58 |
| clarkb | arosen: check both sides too | 22:58 |
| arosen | tcp 0 0 10.0.0.109:39254 10.0.0.91:8888 ESTABLISHED | 22:58 |
| clarkb | jhesketh: ok no worries | 22:58 |
| SpamapS | yolanda: ^^ above I did some odd things with your caching commit.. apologies if it is too aggressive, and your thoughts are very much appreciated. :) | 22:58 |
| clarkb | arosen: check jenkins as well | 22:58 |
| pleia2 | clarkb: do we ever clean up /opt/nodepool ? :) | 22:58 |
| arosen | clarkb: yup it's connected on that too. | 22:58 |
| clarkb | pleia2: that should be the git repo | 22:59 |
| jeblair | clarkb: 158903 references an earlier change; do you know which? | 22:59 |
| arosen | clarkb: hrm I wonder if there is a race condition here if nodes are already "ready" since we need jenkins to get an event first? | 22:59 |
| pleia2 | clarkb: er nodepool_dib/ | 22:59 |
| clarkb | jeblair: sorry, yes one sec | 22:59 |
| clarkb | jeblair: I45732195754e97d287f3c4c3906a48e8d6f13ed7 | 22:59 |
| anteaya | mordred: I'll need jim's comments addressed first please | 22:59 |
| clarkb | arosen: shouldn't be any race conditions | 23:00 |
| *** rm_work|away is now known as rm_work | 23:00 | |
| clarkb | pleia2: arg, dib is leaking I think dib is supposed to clean those dirs up | 23:00 |
| arosen | both jenkins and nodepool are connected together over 8888 | 23:00 |
| clarkb | arosen: in your nodepool log do you see things like 'DEBUG nodepool.NodeUpdateListener: Received: onStarted'? | 23:01 |
| mestery | clarkb: Tahnks for holding that node for me! I'm done for now, I may be back tomorrow to debug again, you can return that one to the pool now. Thanks again! | 23:02 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 23:02 |
| *** harlowja is now known as harlowja_away | 23:02 | |
| clarkb | mestery: ok | 23:03 |
| jeblair | clarkb: that looks like a "nodepool dib is completely broken" error, is that the case? | 23:03 |
| *** bhuvan has joined #openstack-infra | 23:03 | |
| *** bhuvan has joined #openstack-infra | 23:03 | |
| clarkb | jeblair: I don'tthink so because we haven't restarted nodepool since that code merged | 23:03 |
| clarkb | jeblair: the manual commands break because that loads latest code but the daemon code is older so should be working right now | 23:04 |
| jeblair | ah, ok :) | 23:04 |
| clarkb | and unfortunately that happens after the cut out fake testing stuff | 23:04 |
| clarkb | so the tests wouldn't catch it | 23:04 |
| *** harlowja_away is now known as harlowja | 23:05 | |
| *** vilobhmm1 has quit IRC | 23:06 | |
| *** vilobhmm has joined #openstack-infra | 23:06 | |
| arosen | clarkb: nope not yet. | 23:07 |
| arosen | clarkb: I just restarted it though | 23:07 |
| dkehn | zaro: is there a good way to debug the gerrit start to see what is failing? | 23:07 |
| *** bknudson has quit IRC | 23:07 | |
| *** mestery has quit IRC | 23:07 | |
| clarkb | arosen: ok, that should be logged if nodepool receives the events that move nodes to used from ready | 23:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add ansible role for running puppet https://review.openstack.org/158173 | 23:07 |
| arosen | clarkb: k i'll keep an eye for that next them a node goes to ready. | 23:07 |
| clarkb | arosen: it happens when a job starts on a node | 23:08 |
| *** emagana has quit IRC | 23:08 | |
| arosen | clarkb: does nodepool have to talk to jenkins to put a node in a ready state? | 23:08 |
| mordred | jeblair: thanks! that old commit message was from before the rebase - it turns out that between the time I wrote the patch and uploaded it, the os-ansible-deployment folks reworked their things, so I didn't wind up needing to touch their stuff | 23:08 |
| mordred | anteaya: ^^ addressed | 23:08 |
| * anteaya looks again | 23:08 | |
| zaro | dkehn: what is the gerrit start? | 23:09 |
| *** Longgeek has joined #openstack-infra | 23:09 | |
| *** mfink_ has quit IRC | 23:09 | |
| anteaya | ansible galaxy wooo new nomenclature | 23:09 |
| clarkb | arosen: yes, it does so over the jenkins api | 23:09 |
| zaro | dkehn: you can check the log file, maybe even increase the logging level as well. | 23:10 |
| dkehn | zaro: when puppet calls the "/etc/init.d/gerrit start" its returning a 1 instead of 0 | 23:11 |
| clarkb | arosen: basic flow is zuul tells gearman run a job, nodepool see in gearman that there is demand for a job which means it needs to build a node, nodepool builds a node, sshes into it, if that works it adds the node to jenkins, then jenkins via gearman says "I can run that job now" and assigns the job to the node it was given. When the job starts jenkins emits an event that says "job started" when the | 23:11 |
| clarkb | job finishes jenkins emits an event that says "job finished". nodepool receives both events and on start moves the node from ready to used and on finished from used to delete | 23:11 |
| mordred | anteaya: wouldn't want people to be bored | 23:11 |
| dkehn | zaro: would like to zero in on what the issue | 23:11 |
| anteaya | mordred: oh thank you, that is such a danger around here | 23:11 |
| *** shashankhegde has quit IRC | 23:11 | |
| zaro | dkehn: ahh, i guess i would first check the log in review_site/logs/error_log | 23:12 |
| anteaya | mordred: do enlightenten me on why the acl config is ansible-config and the repo is ansible-puppet | 23:12 |
| arosen | clarkb: if i delete a node in jenkins should I see the onStarted message in the log in nodepool? | 23:12 |
| *** Ryan_Lane has quit IRC | 23:12 | |
| clarkb | arosen: no | 23:12 |
| anteaya | are you planning on more repos for which you need to reuse ansible-system | 23:12 |
| clarkb | arosen: only job events are emitted not node state changes | 23:12 |
| anteaya | sorry it is ansible-system.config | 23:12 |
| mordred | anteaya: yes | 23:12 |
| arosen | clarkb: my problem right now is all my nodes are showing up in jenkins but they say: "Ping response time is too long or timed out." | 23:12 |
| mordred | anteaya: I was setting it up to be like the puppet-* repos | 23:12 |
| dkehn | zaro: great thanks, wasn't sure where that would be | 23:13 |
| arosen | I can bring the node back online and they work | 23:13 |
| arosen | i wonder if there was a network blip that caused them to fall all offline in jenkins | 23:13 |
| *** achanda has joined #openstack-infra | 23:13 | |
| arosen | Shouldn't jenkins retry to connect them? It doesn't seem like they ever reconnect. | 23:13 |
| arosen | I can ssh/ping them just fine from the jenkins node. | 23:13 |
| anteaya | mordred: why not reuse one of the other infra acl files | 23:13 |
| anteaya | mordred: I'm sure there is one the same | 23:13 |
| pleia2 | hooray, bare-trusty built on hpcloud-b1 (thanks for your help Hunner and crinkle) | 23:13 |
| pleia2 | doing the other 4 now | 23:14 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/project-config: Add py34 check job for gear https://review.openstack.org/158148 | 23:14 |
| clarkb | arosen: yes jenkins should retry to ssh to them if it loses the connection | 23:14 |
| anteaya | mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack-infra/config.config | 23:14 |
| clarkb | pleia2: woot | 23:14 |
| clarkb | pleia2: I approved the nodepool fix too, so once that is installed we can try image-upload again | 23:14 |
| crinkle | pleia2: \o/ | 23:14 |
| pleia2 | clarkb: great | 23:15 |
| pleia2 | clarkb: then I just have bare-precise, yeah? | 23:15 |
| anteaya | mordred: though that doesn't have tagging | 23:15 |
| arosen | clarkb: so if i stop jenkins and then start it again all my ready nodes in nodepool should also be able to reconnect as well right? | 23:15 |
| clarkb | pleia2: and devstack-precise | 23:15 |
| clarkb | pleia2: devstack-precise is a dib image too, you could start the image-build portion of that node | 23:15 |
| pleia2 | clarkb: ah yes, but that's dib | 23:15 |
| mordred | anteaya: yeah - it seems we tend to do per-repo acl configs in infra | 23:15 |
| pleia2 | clarkb: good idea | 23:15 |
| clarkb | pleia2: then wait on the fix for the uploads | 23:15 |
| * pleia2 nods | 23:16 | |
| anteaya | mordred: we do, okay | 23:16 |
| clarkb | arosen: you should be able to go to each node in jenkins and force reconnection | 23:16 |
| clarkb | arosen: under the node page(s) | 23:16 |
| mordred | anteaya: I don't know that I could tell you why | 23:16 |
| dkehn | zaro: when setting up the contactstore, initially its pointing to review-dev.openstacl.org/fakestore - where should it be pointing to, the head of my gitweb ? | 23:16 |
| arosen | clarkb: that's a lot of clicking on the webpage though? | 23:16 |
| clarkb | arosen: dpends on the number of nodes but very possible | 23:17 |
| arosen | clarkb: yea I have like 100 nodes in there. | 23:17 |
| arosen | clarkb: should I just delete them all in nodepool and let them rebuild? Is that the easiest thing to do? | 23:17 |
| openstackgerrit | Merged openstack-infra/nodepool: Append file type to filename on image upload https://review.openstack.org/158903 | 23:17 |
| *** emagana has joined #openstack-infra | 23:17 | |
| fungi | dkehn: you likely don't need to set the class parameter to enable the contactstore feature | 23:18 |
| clarkb | arosen: I would try force reconnecting one node just to see if that works | 23:18 |
| dkehn | fungi: ok, I will make sure its false | 23:18 |
| clarkb | arosen: if it doesn't then likely an issue with jenkins ssh config | 23:18 |
| fungi | dkehn: that should be conditional in the config template so that it doesn't get included if you set it to false when instantiating the class | 23:18 |
| anteaya | mordred: meh I'm content to cargo cult on this one, I don't think there would be much interest even if I asked the question | 23:18 |
| arosen | clarkb: ah yea that worked and set it to used | 23:18 |
| *** marun has quit IRC | 23:19 | |
| fungi | dkehn: if you don't know where your contactstore server is, then chances are you don't have one (it would be a custom api in some other system for accepting encrypted contact info records from gerrit users) | 23:19 |
| *** chlong has joined #openstack-infra | 23:19 | |
| dkehn | fungi: I didn't touch it, I thought by default it was set to false , but will check, seems like its in the error log | 23:20 |
| *** salv-orl_ has joined #openstack-infra | 23:20 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Utilize dogpile.cache for caching https://review.openstack.org/158906 | 23:20 |
| dkehn | fungi: nope its set to true, crap | 23:20 |
| fungi | dkehn: it should default to false in the gerrit module, but the openstack_infra::gerrit class likely overrides that to true because we use it | 23:21 |
| dkehn | fungi: the system-config/modules/openstack_project/manifests/review_dev.pp was set to true | 23:21 |
| fungi | dkehn: ahh, yep. that's the class for our review-dev.openstack.org server | 23:22 |
| *** salv-orlando has quit IRC | 23:22 | |
| *** dangers is now known as dangers_away | 23:22 | |
| fungi | dkehn: we enable it there and point it to a local cgi which implements a dumb fake of the api so that we can test the feature | 23:22 |
| dkehn | fungi: tahats the one I'm using does that sound correct, just creating a local.pp in the system-config/manifest and kicking it | 23:23 |
| *** baoli has quit IRC | 23:23 | |
| *** baoli has joined #openstack-infra | 23:23 | |
| *** hichihara has joined #openstack-infra | 23:24 | |
| fungi | dkehn: you might consider copying it instead. unless you really want to run a copy of review-dev.openstack.org (the openstack_project::gerrit class is intended to be what you would use as a more generic class and then we build on that with review.pp and review_dev.pp there_ | 23:25 |
| reed | mordred, anteaya: I remember one of you at some point was investigating an object storage system for the Board, owncloud and something else. IIRC nothing 'good enough' was found. Is the result of the investigation written down somewhere? | 23:25 |
| *** sdake_ has joined #openstack-infra | 23:25 | |
| *** cody-somerville has quit IRC | 23:25 | |
| mordred | reed: no, I don't think it is - I think the biggest problem with owncloud when we looked in to it was that its swift support was very nascent | 23:25 |
| mordred | reed: it's entirely possible that it's gotten better now | 23:26 |
| anteaya | I couldn't completely automate the deployement | 23:26 |
| fungi | reed: i believe the other main hurdle with any of the things which were evaluated was that they were all web apps which didn't have any ability to automate installation. they all assumed some human manually entering configuration into a web interface | 23:26 |
| reed | mordred, k, thanks. I'm trying to write a follow up to Lauren's message | 23:26 |
| fungi | yeah, what anteaya said | 23:26 |
| anteaya | there was always a step where I had to log in via a browser which was verboten | 23:26 |
| anteaya | but it has been at least a year since I took a look at it | 23:27 |
| reed | gotcha | 23:27 |
| mordred | yeah. it's the classic "we'll make it easier for you" which makes it harder | 23:27 |
| *** dimtruck is now known as zz_dimtruck | 23:27 | |
| fungi | web app managing its own config files paradigm | 23:27 |
| fungi | which often makes separately managing those config files hard/impossible | 23:27 |
| *** gyee has joined #openstack-infra | 23:27 | |
| reed | user friendliness :) | 23:27 |
| anteaya | reed: if it has changed, then lead me to docs and I can try again | 23:28 |
| reed | anteaya, last time I checked owncloud I barfed | 23:28 |
| reed | but I'll try again | 23:28 |
| anteaya | reed: well I don't consider that a ringing endorsement | 23:28 |
| anteaya | if that happens again just don't tell me, okay? | 23:28 |
| *** andreykurilin_ has quit IRC | 23:28 | |
| reed | :) | 23:29 |
| anteaya | I tried owncloud and I tried seafile | 23:29 |
| dkehn | fungi: so I'm looking at the system-config/manifest/site.pp nnot finding the openstack_infra::gerrt | 23:29 |
| fungi | dkehn: i mistyped the first time, it's openstack_project::gerrit but you won't see it in site.pp. it gets instantiated by openstack_project::review and openstack_project::review_dev | 23:30 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Add spec for reusable OpenStack CI solution https://review.openstack.org/139745 | 23:30 |
| fungi | dkehn: it's basically the more generalized class which our production and dev gerrit servers both use | 23:30 |
| clarkb | pliea2 if you `sudo puppet agent --test` on nodepool.o.o you should get the latest nodepool and do image uploads | 23:30 |
| dkehn | fungi: ok, so when I do puppet apply --debug -l /tmp/manifest.log --modulepath=modules:/etc/puppet/modules manifests/local.pp and local has review-dev copied into it doe sthat make sense | 23:30 |
| clarkb | seriously I type bad | 23:30 |
| fungi | dkehn: alternatively, you can simply look at it for an example of how to directly drive our puppet module for gerrit (the openstack-infra/puppet-gerrit repo) | 23:31 |
| Clint | 1/win 23 | 23:31 |
| * Clint sighs. | 23:31 | |
| fungi | dkehn: yeah, that should probably do what you want | 23:31 |
| fungi | dkehn: and then edit the things in local.pp you want to change from what was in the review_dev.pp you copied | 23:32 |
| dkehn | fungi: understand, all I know is I want to get gerrit up and running so I can try and configure it to my git repo, so I'm not sure what I want | 23:33 |
| *** gordc has quit IRC | 23:33 | |
| dkehn | fungi: really the only thing that is un-commented in the lcoal.pp is the mysql details | 23:33 |
| mordred | Clint: you win | 23:34 |
| clarkb | 23 somethings | 23:34 |
| Clint | \o/ | 23:34 |
| Clint | that's the second time i did that today | 23:34 |
| fungi | dkehn: if all you _really_ want is a running gerrit server and you're not interested in puppet, you can just install gerrit manually (their install docs are pretty short). it's also a good way to get a feel for what the puppet module and the classes using it actually do | 23:34 |
| mordred | for the record, I just started talking to someone in window 285 | 23:35 |
| Clint | this morning i did "0/win 60" in a work channel | 23:35 |
| Clint | it must be early dementia | 23:35 |
| fungi | it's all full of win | 23:35 |
| clarkb | Clint: switch to meta-j + numbers | 23:36 |
| clarkb | Clint: never /win again | 23:36 |
| * Clint squints. | 23:36 | |
| Clint | i use ESC-$num for 1-19 | 23:36 |
| clarkb | oh must be irssi | 23:37 |
| Clint | yup | 23:37 |
| clarkb | I think irssi has a plugin to do meta-j like weechat | 23:37 |
| Clint | interesting | 23:37 |
| dkehn | fungi: yeha did that and got it up, was just trying to get more of the modules and pluging that are provided with openstack approach | 23:37 |
| clarkb | but basically meta-j then two numbers to switch windows | 23:37 |
| fungi | dkehn: okay, cool. well, sounds like you're on your way down a good path then | 23:38 |
| *** erikmwilson has quit IRC | 23:39 | |
| *** erikmwil- is now known as erikmwilson | 23:39 | |
| Clint | clarkb: found one for /## but not what you're saying | 23:42 |
| *** Longgeek has quit IRC | 23:42 | |
| *** ayoung has quit IRC | 23:42 | |
| jeblair | irssi for the /win | 23:42 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add unit tests for meta module https://review.openstack.org/158915 | 23:45 |
| SpamapS | jeblair: irssi would you did there | 23:45 |
| *** emagana has quit IRC | 23:47 | |
| *** emagana has joined #openstack-infra | 23:48 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/nodepool: Reset the client object before creating the image https://review.openstack.org/74943 | 23:48 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add unit tests for meta module https://review.openstack.org/158915 | 23:48 |
| *** achanda has quit IRC | 23:48 | |
| clarkb | jhesketh: commented on the globbing change for zuul swift uploads | 23:50 |
| *** chlong_ has joined #openstack-infra | 23:50 | |
| clarkb | jhesketh: need to also get the nodepool dib elements updated | 23:50 |
| clarkb | jhesketh: the rest of the stack lgtm | 23:50 |
| *** emagana has quit IRC | 23:52 | |
| *** achanda has joined #openstack-infra | 23:53 | |
| anteaya | okay reed, I will try again, I make no promises: https://doc.owncloud.org/server/8.0/admin_manual/configuration/automatic_configuration.html | 23:53 |
| *** mriedem has joined #openstack-infra | 23:53 | |
| mordred | SpamapS: is making me happy | 23:53 |
| *** ashleighfarnham has quit IRC | 23:54 | |
| mordred | reed, anteaya: "The ../owncloud/config/autoconfig.php is automatically removed after the initial configuration has been applied." STAB STAB STAB | 23:55 |
| mordred | it's like they've never heard of config management ever | 23:55 |
| reed | anteaya, before you venture there again, have you tried the basic functionalities of owncloud to see if they satisfy our needs? | 23:55 |
| *** ZZelle_ has quit IRC | 23:56 | |
| anteaya | reed: it I can't configure it automatically it doesn't matter | 23:56 |
| reed | I ask because last time I checked the filesharing mechanism was non-existent (or at least hidden very well) | 23:56 |
| anteaya | and no, other than being able to deploy via scripts, taht is the only need I have | 23:56 |
| *** salv-orlando has joined #openstack-infra | 23:56 | |
| anteaya | reed: okay yeah well if the app doesn't meet your needs, nor mine, I'm not up for wasting my time | 23:57 |
| anteaya | reed: do let me know if you find something that might work | 23:57 |
| jhesketh | clarkb: cool, thanks | 23:57 |
| reed | anteaya, will do | 23:57 |
| *** prad has quit IRC | 23:57 | |
| *** salv-orl_ has quit IRC | 23:57 | |
| *** bdpayne has quit IRC | 23:58 | |
| anteaya | mordred: I wanted to see SpamapS in a conversation with Clint | 23:58 |
| pleia2 | clarkb: that patch seems to have done it, uploading devstack-trusty to hpcloud-b1 now | 23:58 |
| anteaya | that would make me happy | 23:58 |
| mordred | anteaya: I know. it's the only thing I realy want in life | 23:59 |
| *** baoli has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
