| fungi | grey area, i could go either way on the project creators guide | 00:00 |
|---|---|---|
| *** hdd has joined #openstack-infra | 00:00 | |
| fungi | it's just nice to have great docs. i'm so glad the problem is not "why don't we have documentation" now, it's "where do we put it?" | 00:00 |
| jogo | jeblair: thanks! This is part of larger effort to make the docs.openstack.org/developer/nova/ useable | 00:01 |
| jeblair | fungi: haha! indeed | 00:01 |
| *** mahito has joined #openstack-infra | 00:02 | |
| fungi | first world developer problems | 00:02 |
| jeblair | jogo: cool, thanks :) | 00:02 |
| jeblair | i need to shift my position in space; bye for now | 00:02 |
| fungi | if you shift it particularly, watch out for time dilation | 00:03 |
| fungi | er, particularly quickly | 00:03 |
| *** hdd has quit IRC | 00:04 | |
| openstackgerrit | Merged openstack/requirements: Use httpretty < 0.8.7 to avoid broken unittests https://review.openstack.org/182340 | 00:04 |
| fungi | (and also lorentz contraction i suppose) | 00:04 |
| *** banix has joined #openstack-infra | 00:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-elasticsearch: Cleanup old ES logs https://review.openstack.org/182895 | 00:05 |
| clarkb | pleia2: jesusaurus (and others) ^ that should keep ES logs reasonable | 00:06 |
| *** teran_ has quit IRC | 00:06 | |
| fungi | grr. we still don't have any jobs registered for bare-precise workers | 00:06 |
| clarkb | fungi: are they just failing to boot? | 00:06 |
| harlowja | qq, i'm seeing | 00:07 |
| fungi | there's 1 in use for more than an hour, and one stuck in building for almost as long | 00:07 |
| harlowja | 'NOT_REGISTERED' on https://review.openstack.org/#/c/181581/ for 2.6; trying a recheck, did something else get bused that might have removed that? | 00:07 |
| harlowja | *busted | 00:07 |
| *** teran has joined #openstack-infra | 00:08 | |
| fungi | i was able to delete the building one, so hopefully it will be replaced shorting by one that successfully boots | 00:08 |
| fungi | harlowja: any python 3.6 jobs should be fune to recheck now. those have been successfully registered | 00:09 |
| harlowja | k | 00:09 |
| harlowja | thx | 00:09 |
| fungi | we're just left dealing with the bare-precise jobs now | 00:09 |
| harlowja | okie dokie | 00:09 |
| fungi | okay, there's a new one building. here's crossed fingers | 00:10 |
| fungi | there's something not right about the test nodes graph | 00:11 |
| fungi | looks like it flatlined | 00:12 |
| harlowja | CPR CPR | 00:12 |
| fungi | i think it's misreporting | 00:12 |
| clarkb | fungi: ya I wonder if it stopped sending statsd for some reason | 00:12 |
| *** sputnik13 has quit IRC | 00:12 | |
| fungi | okay, that bare-precise worker went ready, so we should have jobs in a sec | 00:12 |
| clarkb | fungi: iirc you can just tcpdump -A the udp port but I Have to remember what port that is | 00:13 |
| fungi | looks like they've registered now | 00:13 |
| clarkb | lol did you guys like the clarkb not in sudoers from the git-fe* nodes? | 00:13 |
| clarkb | that happened because i didn't regroup after changing the sudoer group | 00:14 |
| fungi | boris-42: you should be safe to recheck for not_registered jobs which should have run on bare-precise workers | 00:14 |
| pleia2 | clarkb: I saw, figured it was something like that :) | 00:14 |
| *** gyee has quit IRC | 00:14 | |
| fungi | clarkb: yeah, i ignored it. i knew you were on there and doing stuff | 00:14 |
| clarkb | fungi: `sudo tcpdump -i any -A port 8125` doesn't show anything on nodepool.o.o | 00:15 |
| clarkb | I think that is the correct port | 00:15 |
| fungi | clarkb: now that we've got stuff built and registered i can restart nodepoold | 00:15 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Improve error message on auth_plugin failure https://review.openstack.org/182898 | 00:16 |
| SpamapS | greghaynes: ^ trying to get more insight into what is failing | 00:16 |
| fungi | the graphite server seems healthy | 00:16 |
| clarkb | fungi: ya I think it is nodepool side based on my tcpdump | 00:16 |
| clarkb | /etc/default/nodepool sets up STATSD | 00:17 |
| fungi | clarkb: when did nodepool get restarted for the shade patch? | 00:17 |
| clarkb | let me check irc | 00:18 |
| clarkb | 2041ish | 00:18 |
| clarkb | probably a couple minutes before that | 00:18 |
| clarkb | fungi: looking at the environ file for that process in /proc shows that the stuff was passed in | 00:19 |
| clarkb | fungi: doesn't appear related to when I restarted nodepool for shade if I am reading this graph correctly | 00:20 |
| fungi | yeah, the zuul job queue says there are ~300 jobs running, while the test nodes graph says maybe 75 used | 00:20 |
| clarkb | graphite.o.o was not rebooted | 00:21 |
| *** haypo has joined #openstack-infra | 00:21 | |
| fungi | looks like your nodepool restart was a good couple hours before the graph went weird, so more likely this is from our reboots | 00:21 |
| clarkb | I wonder if when doing a hard reboot networking is such that we couldn't resolve that node or hit it so statsd just turns off | 00:21 |
| fungi | maybe | 00:22 |
| fungi | i'm not seeing anything in the logs to explain it | 00:22 |
| *** ashleighfarnham has quit IRC | 00:23 | |
| haypo | jenkins voted -1 on a change, but all tests pass: https://review.openstack.org/#/c/182877/ there is "gate-oslo.db-python26 NOT_REGISTERED" which looks suspicious. is there something wrong with this gate? | 00:23 |
| *** ashleighfarnham has joined #openstack-infra | 00:23 | |
| bknudson | keystoneclient functional tests are failing... http://logs.openstack.org/53/182453/1/gate/gate-keystoneclient-dsvm-functional/cb26021/console.html | 00:25 |
| bknudson | looks like for some reason the env vars aren't making it to the tox job | 00:25 |
| clarkb | fungi: https://github.com/jsocol/pystatsd/blob/master/statsd/client.py#L134-L143 it definitely appears to only do the getaddrinfo when it starts | 00:25 |
| *** pabelanger has quit IRC | 00:26 | |
| fungi | bknudson: if it's complaining about envbindir then new tox is not your friend and you need to do something like neutron did to work around it (or you are not going to space today) | 00:26 |
| *** mahito has quit IRC | 00:26 | |
| fungi | haypo: recheck. when we restarted zuul some jobs ran before any new bare-centos6 workers had registered their ability to run python 2.6 jobs | 00:26 |
| clarkb | bknudson: new tox filters out env vars, you can either whitelist them or explicitly setenv in your tox.ini | 00:26 |
| bknudson | new tox? | 00:26 |
| bknudson | ahh | 00:26 |
| fungi | 2.0~ | 00:27 |
| fungi | s/~/!/ | 00:27 |
| haypo | fungi, hum, ok :) | 00:27 |
| bknudson | clarkb: fungi: thanks, I'll look into it. | 00:27 |
| fungi | clarkb: yep, definitely no packets. i agree that nodepoold probably started before the vm had good network | 00:27 |
| *** ashleighfarnham has quit IRC | 00:28 | |
| fungi | restarting nodepoold now unless there's anything else you want to analyze first | 00:28 |
| fungi | should it have thrown an error about that initially? | 00:28 |
| *** dangers is now known as dangers_away | 00:28 | |
| fungi | i couldn't find one if so | 00:28 |
| clarkb | fungi: reading further into statsd code I am convinced the issue was in name resolution | 00:29 |
| clarkb | fungi: I don't think I need to read anything else | 00:29 |
| clarkb | fungi: no statsd masks the error for you | 00:29 |
| zaro | linuxmaniac: finally got around to looking at https://review.openstack.org/#/c/157735/ | 00:29 |
| fungi | i expected as much | 00:29 |
| clarkb | see https://github.com/jsocol/pystatsd/blob/v2.1.2/statsd/__init__.py#L43 | 00:29 |
| clarkb | fungi: and of course we can't manipulate sysv init order when using upstart and puppet | 00:30 |
| *** mahito has joined #openstack-infra | 00:30 | |
| fungi | yep, i agree | 00:30 |
| fungi | okay, restarting the daemon now | 00:30 |
| *** zz_dimtruck is now known as dimtruck | 00:30 | |
| clarkb | I am checking zuul.o.o now | 00:30 |
| *** haypo has quit IRC | 00:31 | |
| clarkb | I see packets on zuul.o.o | 00:31 |
| *** otter768 has joined #openstack-infra | 00:31 | |
| fungi | clarkb: also when you have a sec, https://review.openstack.org/182890 should hopefully fix puppet on the zuul servers | 00:31 |
| clarkb | fungi: looking | 00:32 |
| clarkb | approved, thats a simple change | 00:33 |
| fungi | it was a simple bug ;) | 00:33 |
| clarkb | I am going to go visit family now, will check in on es cluster later | 00:33 |
| fungi | have a good evening | 00:33 |
| bknudson | easy fix -- https://review.openstack.org/182900 | 00:33 |
| fungi | i'm winding down finally as well | 00:33 |
| fungi | bknudson: oh, that's a different bug then. i take it also new for tox 2.0 | 00:34 |
| fungi | bknudson: if passenv is more recent than tox 1.6 you probably also want to make sure to bump minversion at the top of the file (i've not seen it used but could predate 1.6 for all i know) | 00:35 |
| bknudson | fungi: I can try it with an older tox to see if tox barfs on it. | 00:35 |
| *** bhuvan has quit IRC | 00:35 | |
| *** otter768 has quit IRC | 00:36 | |
| *** Alexandra_ has joined #openstack-infra | 00:37 | |
| bknudson | seems like tox should ignore directives it doesn't understand | 00:37 |
| fungi | seems like a lot of things tox should do... i don't have much faith in assumptions about its behaviors any longer | 00:37 |
| fungi | well, the test nodes graph is once again nonlinear. that seems to have brought its stats reporting back to reality again | 00:39 |
| *** ZZelle has joined #openstack-infra | 00:39 | |
| *** claudiub has quit IRC | 00:41 | |
| *** rbradfor has quit IRC | 00:42 | |
| *** mahito has quit IRC | 00:42 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:43 | |
| *** mahito has joined #openstack-infra | 00:44 | |
| *** teran_ has joined #openstack-infra | 00:45 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul: Make sure /var/lib/zuul/www/lib exists https://review.openstack.org/182890 | 00:45 |
| *** teran has quit IRC | 00:45 | |
| *** mtanino has joined #openstack-infra | 00:46 | |
| *** tiswanso has joined #openstack-infra | 00:49 | |
| bknudson | tox docs still say {envbindir} -- http://tox.readthedocs.org/en/latest/config.html#substitutions-for-virtualenv-related-sections | 00:50 |
| fungi | yeah, the manual claims it exists. i meant to open a bug report about that and then got sidetracked by so so many vm reboots | 00:51 |
| fungi | if you beat me to it. i won't complain | 00:51 |
| openstackgerrit | Merged openstack-infra/system-config: Move security.openstack.org to HTTPS https://review.openstack.org/155099 | 00:57 |
| openstackgerrit | Merged openstack-infra/system-config: Correct ci.o.o redirects so slashes are optional https://review.openstack.org/182790 | 01:00 |
| *** alexsyip has quit IRC | 01:04 | |
| bknudson | fungi: https://bitbucket.org/hpk42/tox/issue/246/envbindir-substitution-key-not-found | 01:06 |
| *** _1_Abed has joined #openstack-infra | 01:07 | |
| *** ZZelle has quit IRC | 01:07 | |
| *** _1_Abed has left #openstack-infra | 01:08 | |
| openstackgerrit | Debo~ Dutta proposed openstack-infra/project-config: WIP: Cognitive - Machine Learning as a Service https://review.openstack.org/182886 | 01:09 |
| *** mattfarina has joined #openstack-infra | 01:13 | |
| *** hdd has joined #openstack-infra | 01:14 | |
| *** mattfarina has quit IRC | 01:16 | |
| *** tiswanso has quit IRC | 01:18 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:18 | |
| *** tiswanso has joined #openstack-infra | 01:18 | |
| *** puranamr has quit IRC | 01:19 | |
| *** zhiwei has joined #openstack-infra | 01:20 | |
| fungi | bknudson: awesome--thanks!!! | 01:25 |
| bknudson | no problem. | 01:26 |
| *** ivar-lazzaro has quit IRC | 01:26 | |
| bknudson | looks like it's only some nodes that have tox 2 | 01:26 |
| openstackgerrit | Debo~ Dutta proposed openstack-infra/project-config: WIP: Cognitive - Machine Learning as a Service https://review.openstack.org/182886 | 01:26 |
| fungi | it'll depend on what images were updated and what weren't | 01:27 |
| fungi | we probably have some stale images for some labels in some providers | 01:27 |
| fungi | okay, i've confirmed that security.o.o is properly redirecting to https and the ssl configuration looks good (chain is intact and whatnot) | 01:36 |
| fungi | also i've switched the dns for ci.openstack.org to our redirector now, so everyone should be ending up at docs.openstack.org/infra/something when they try to follow old links or bookmarks to it | 01:37 |
| fungi | looks like it's propagated and working as expected | 01:37 |
| openstackgerrit | Debo~ Dutta proposed openstack-infra/project-config: WIP: Cognitive - Machine Learning as a Service https://review.openstack.org/182886 | 01:37 |
| *** Longgeek has quit IRC | 01:38 | |
| fungi | also that puppet-zuul fix seems to have cleared up the puppetboard failures list now | 01:39 |
| *** woodster_ has quit IRC | 01:40 | |
| *** dims has quit IRC | 01:42 | |
| *** Krinkle is now known as Krinkle|detached | 01:44 | |
| *** mahito has quit IRC | 01:44 | |
| *** Rockyg has quit IRC | 01:47 | |
| *** banix has quit IRC | 01:48 | |
| *** teran has joined #openstack-infra | 01:56 | |
| *** teran_ has quit IRC | 01:58 | |
| *** banix has joined #openstack-infra | 02:08 | |
| *** sdake has quit IRC | 02:09 | |
| *** mahito has joined #openstack-infra | 02:09 | |
| *** Alexandra_ has quit IRC | 02:10 | |
| *** packet has quit IRC | 02:12 | |
| *** sdake has joined #openstack-infra | 02:12 | |
| fungi | brb. time to reboot my own servers! | 02:14 |
| *** fungi has quit IRC | 02:14 | |
| *** spzala has joined #openstack-infra | 02:16 | |
| *** yamahata has quit IRC | 02:16 | |
| *** packet has joined #openstack-infra | 02:19 | |
| *** sarob has quit IRC | 02:22 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Remove sahara from the default configuration https://review.openstack.org/181230 | 02:25 |
| openstackgerrit | Merged openstack/requirements: Removing unused irdragdrop from requirements https://review.openstack.org/176568 | 02:25 |
| *** enikanorov2 has quit IRC | 02:27 | |
| *** unicell has quit IRC | 02:27 | |
| *** fungi has joined #openstack-infra | 02:27 | |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras and env markers https://review.openstack.org/181795 | 02:28 |
| *** Longgeek has joined #openstack-infra | 02:28 | |
| tchaypo | so that time, git review took | 02:31 |
| tchaypo | real4m9.517s | 02:31 |
| tchaypo | user0m0.591s | 02:31 |
| fungi | that's a lotta waitin' | 02:32 |
| tchaypo | 1m53s of that happened *after* the bot announced here that I’d proposed something | 02:32 |
| *** otter768 has joined #openstack-infra | 02:32 | |
| *** yamamoto has joined #openstack-infra | 02:35 | |
| *** patrickeast has quit IRC | 02:35 | |
| *** jeblair has quit IRC | 02:36 | |
| *** jeblair has joined #openstack-infra | 02:36 | |
| *** otter768 has quit IRC | 02:37 | |
| *** asadoughi has quit IRC | 02:40 | |
| *** Longgeek has quit IRC | 02:41 | |
| *** Longgeek_ has joined #openstack-infra | 02:41 | |
| *** asadoughi has joined #openstack-infra | 02:44 | |
| *** baoli has joined #openstack-infra | 02:48 | |
| *** bhunter71 has quit IRC | 02:53 | |
| *** enikanorov2 has joined #openstack-infra | 02:57 | |
| *** Longgeek_ has quit IRC | 03:04 | |
| *** ivar-lazzaro has joined #openstack-infra | 03:05 | |
| *** alex_xu has quit IRC | 03:05 | |
| *** Longgeek has joined #openstack-infra | 03:06 | |
| *** blahdeblah has joined #openstack-infra | 03:07 | |
| *** david-lyle_ has quit IRC | 03:11 | |
| *** david-lyle_ has joined #openstack-infra | 03:12 | |
| *** panda has quit IRC | 03:13 | |
| *** panda has joined #openstack-infra | 03:14 | |
| *** hdd has quit IRC | 03:15 | |
| *** david-lyle_ has quit IRC | 03:17 | |
| *** markvoelker has joined #openstack-infra | 03:18 | |
| *** achanda has quit IRC | 03:23 | |
| *** banix has quit IRC | 03:24 | |
| SpamapS | greghaynes: looks like it was something that got fixed elsewhere... tests are passing now | 03:26 |
| greghaynes | huh... | 03:26 |
| greghaynes | :( | 03:27 |
| *** mahito has quit IRC | 03:27 | |
| greghaynes | I have a devstack I stood up during that time period, I might see if I can repro | 03:27 |
| *** Alexandra_ has joined #openstack-infra | 03:27 | |
| *** Alexandra_ has quit IRC | 03:28 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 03:28 | |
| *** Alexandra_ has joined #openstack-infra | 03:28 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 03:28 | |
| SpamapS | greghaynes: http://logs.openstack.org/98/182898/1/check/check-shade-dsvm-functional-neutron/d16eecf/console.html#_2015-05-14_00_35_33_476 | 03:28 |
| SpamapS | 2015-05-14 00:35:33.506 | 2015-05-14 00:35:33.480 | shade.exc.OpenStackCloudException: Error getting compute endpoint: Error constructing auth plugin: password __init__() takes at least 2 arguments (1 given) | 03:29 |
| SpamapS | I wonder if keystoneclient had some.. issues? | 03:29 |
| greghaynes | oh, yea | 03:29 |
| jamielennox | i don't think keystoneclient changed anything | 03:30 |
| greghaynes | :/ | 03:30 |
| SpamapS | greghaynes: http://logs.openstack.org/45/182845/1/check/check-shade-dsvm-functional-neutron/6272089/ this one succeeded | 03:30 |
| *** dims has joined #openstack-infra | 03:31 | |
| SpamapS | could also still be os-client-config issues | 03:31 |
| *** packet has quit IRC | 03:31 | |
| *** dimtruck is now known as zz_dimtruck | 03:31 | |
| * SpamapS goes back to family time | 03:32 | |
| *** mahito has joined #openstack-infra | 03:32 | |
| *** tiswanso has quit IRC | 03:32 | |
| *** tnovacik has joined #openstack-infra | 03:32 | |
| jamielennox | greghaynes, SpamapS: there is a tox issues with v2 | 03:32 |
| greghaynes | yea, I dont think theres a whole lot we can do with that amount of info :( | 03:32 |
| greghaynes | jamielennox: hrm | 03:33 |
| jamielennox | with tox 2 you need to whitelist ENV vars that are passed through | 03:33 |
| jamielennox | i assume shade is testing that way as most other functional tests do | 03:33 |
| greghaynes | jamielennox: for the auth vars? | 03:33 |
| *** btully has quit IRC | 03:34 | |
| jamielennox | greghaynes: for everything | 03:34 |
| greghaynes | huh, thats definitely plausible | 03:34 |
| jamielennox | if it's doing functional tests by sourcing the rc file that comes from devstack it might be a problem | 03:34 |
| jamielennox | https://review.openstack.org/#/c/182900/ is the ksc fix | 03:34 |
| greghaynes | jamielennox: were testing a thing that loads the config - os-client-config | 03:34 |
| greghaynes | but it would be succeptible to the same issue | 03:34 |
| jamielennox | but it's only in tox 2 so not all the CI tests are picking it up | 03:34 |
| clarkb | are the vars laoded before or after running tox? | 03:36 |
| clarkb | if before then yes likely the issue | 03:36 |
| greghaynes | I actually have no idea for that specific test | 03:36 |
| greghaynes | os-c-c supports both | 03:36 |
| clarkb | Probably makes sense to write a config fike instead | 03:36 |
| clarkb | *file | 03:36 |
| *** dims has quit IRC | 03:36 | |
| *** baoli has quit IRC | 03:37 | |
| jamielennox | there's a few things we could fix up from the accrc files that devstack writes | 03:38 |
| *** BharatK has joined #openstack-infra | 03:39 | |
| greghaynes | im working on some stuff right now to get devstack writing out an os-cloud-config config file | 03:40 |
| greghaynes | so, we could probably just use that | 03:40 |
| greghaynes | er, os-client-config | 03:40 |
| *** packet has joined #openstack-infra | 03:41 | |
| *** coolsvap|afk is now known as coolsvap | 03:42 | |
| *** yamahata has joined #openstack-infra | 03:46 | |
| *** banix has joined #openstack-infra | 03:46 | |
| *** tnovacik has quit IRC | 03:47 | |
| *** banix has quit IRC | 03:47 | |
| *** yamamoto has quit IRC | 03:48 | |
| *** yamamoto_ has joined #openstack-infra | 03:48 | |
| *** Longgeek has quit IRC | 03:50 | |
| *** mahito has quit IRC | 03:53 | |
| *** Longgeek has joined #openstack-infra | 03:55 | |
| *** mahito has joined #openstack-infra | 03:58 | |
| timrc | clarkb, jeblair, mordred, fungi I've hit a bug using apache::vhost::proxy provided by the 0.0.4 apache puppet module on Trusty with default apache 2.4.7-1ubuntu4.4. Basically because the site file is not written out to sites-enabled with a .conf terminating it, the vhost goes unrecognized. The fix was introduced in 0.3.0 -- https://github.com/puppetlabs/puppetlabs-apache/commit/dde8570287f03b3862b75ee6b878fbcd20512ac6 | 03:58 |
| nibalizer | timrc: propose a fix to openstackinfra-httpd ? | 04:00 |
| clarkb | ir just use vhost | 04:00 |
| StevenK | timrc: The apache2 configuration was changed between Lucid and Trusty to stop doing '*' and start doing '*.conf' | 04:00 |
| clarkb | thats why we sticj to the old version | 04:00 |
| clarkb | vhost just works | 04:00 |
| *** Longgeek has quit IRC | 04:00 | |
| clarkb | gerrit, nodepool, and zuul are all trusty at thus point with apache2.4 too | 04:02 |
| timrc | clarkb, Cool. Surprisingly simple solution to my problem :) | 04:03 |
| timrc | clarkb, Thanks, man! | 04:03 |
| * timrc just works around the bugs :) | 04:04 | |
| *** jamespage has quit IRC | 04:05 | |
| *** gnuoy has quit IRC | 04:05 | |
| *** jamespage has joined #openstack-infra | 04:06 | |
| *** gnuoy has joined #openstack-infra | 04:06 | |
| *** yamamoto_ has quit IRC | 04:06 | |
| *** amotoki_ has joined #openstack-infra | 04:07 | |
| *** spzala has quit IRC | 04:10 | |
| *** mrmartin has joined #openstack-infra | 04:12 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/infra-specs: Jenkins Job Builder 2.0.0 API Changes/Rewrite https://review.openstack.org/182542 | 04:12 |
| *** dannywilson has quit IRC | 04:14 | |
| *** unicell has joined #openstack-infra | 04:16 | |
| *** packet has quit IRC | 04:18 | |
| *** achanda has joined #openstack-infra | 04:20 | |
| *** unicell has quit IRC | 04:21 | |
| *** unicell has joined #openstack-infra | 04:22 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Change defcore / refstack Chair https://review.openstack.org/182925 | 04:24 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Update Chef cookbooks meeting https://review.openstack.org/182926 | 04:24 |
| *** JoshNang_ has joined #openstack-infra | 04:26 | |
| *** JoshNang_ has quit IRC | 04:27 | |
| *** JoshNang_ has joined #openstack-infra | 04:27 | |
| waynr | is anyone aware of JJB config repositories other than openstack-infra/project-configs and wikimedia's? | 04:27 |
| *** Sukhdev has joined #openstack-infra | 04:29 | |
| *** amotoki_ has quit IRC | 04:31 | |
| nibalizer | waynr: I have a private one ... | 04:31 |
| nibalizer | so if you have questions about it I can answer them | 04:31 |
| *** JoshNang_ has quit IRC | 04:31 | |
| nibalizer | not super useful I know | 04:31 |
| *** JoshNang_ has joined #openstack-infra | 04:31 | |
| waynr | i was mostly curious if there were other open source repos, but thanks for the offer :) | 04:32 |
| *** hdd has joined #openstack-infra | 04:32 | |
| waynr | i am curious to see what kind of JJB features are widely used | 04:32 |
| waynr | the 500+ jobs that I've been managing for example don't use macros at all, but I see they are fairly widely used in openstack-infra/project-configs | 04:33 |
| *** otter768 has joined #openstack-infra | 04:33 | |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla https://review.openstack.org/182927 | 04:34 |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla https://review.openstack.org/182927 | 04:34 |
| *** btully has joined #openstack-infra | 04:37 | |
| *** yamamoto has joined #openstack-infra | 04:37 | |
| *** otter768 has quit IRC | 04:38 | |
| *** BharatK has quit IRC | 04:40 | |
| *** btully has quit IRC | 04:41 | |
| *** mrmartin has quit IRC | 04:41 | |
| *** woodster_ has joined #openstack-infra | 04:47 | |
| *** ppai has joined #openstack-infra | 04:47 | |
| *** markvoelker has quit IRC | 04:47 | |
| *** JoshNang_ has quit IRC | 04:49 | |
| *** markvoelker has joined #openstack-infra | 04:49 | |
| *** teran has quit IRC | 04:50 | |
| *** sks has joined #openstack-infra | 04:52 | |
| *** sarob has joined #openstack-infra | 04:52 | |
| *** BharatK has joined #openstack-infra | 04:53 | |
| *** deepakcs has joined #openstack-infra | 04:54 | |
| *** Longgeek has joined #openstack-infra | 04:57 | |
| *** Swami_ has quit IRC | 04:57 | |
| *** yfried has quit IRC | 04:59 | |
| *** abregman has joined #openstack-infra | 05:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 05:01 |
| *** nfedotov has joined #openstack-infra | 05:01 | |
| ppai | Hi guys, I'm not able to push signed tags to a stackforge project in which I'm part of the release group. http://paste.openstack.org/show/222457/ Is there something that I'm doing wrong ? | 05:01 |
| ppai | *unsigned annotated tags | 05:02 |
| clarkb | they must be signed | 05:02 |
| *** alex_xu has joined #openstack-infra | 05:03 | |
| ppai | clarkb, thanks! | 05:03 |
| *** btully has joined #openstack-infra | 05:04 | |
| *** turul_ has quit IRC | 05:07 | |
| *** Longgeek has quit IRC | 05:08 | |
| *** sarob has quit IRC | 05:14 | |
| *** pal has joined #openstack-infra | 05:17 | |
| *** mahito has quit IRC | 05:20 | |
| *** toabctl has quit IRC | 05:21 | |
| *** sdake_ has joined #openstack-infra | 05:22 | |
| *** jlvillal has quit IRC | 05:26 | |
| *** sdake has quit IRC | 05:26 | |
| *** BharatK has quit IRC | 05:30 | |
| *** naggappan has joined #openstack-infra | 05:36 | |
| *** sdake_ is now known as sdake | 05:38 | |
| *** marun_ has joined #openstack-infra | 05:38 | |
| *** marun has quit IRC | 05:38 | |
| *** ildikov has quit IRC | 05:41 | |
| *** hdd has quit IRC | 05:41 | |
| *** JoshNang_ has joined #openstack-infra | 05:41 | |
| *** BharatK has joined #openstack-infra | 05:42 | |
| *** JoshNang_ has quit IRC | 05:45 | |
| *** JoshNang_ has joined #openstack-infra | 05:45 | |
| *** smccully has quit IRC | 05:45 | |
| *** smccully has joined #openstack-infra | 05:46 | |
| *** claudiub has joined #openstack-infra | 05:47 | |
| *** pal has quit IRC | 05:47 | |
| *** baoli has joined #openstack-infra | 05:48 | |
| *** nfedotov has quit IRC | 05:49 | |
| naggappan | hi i installed the gerrit and its UI is different from the one which is used in "review.openstack.org". In review.openstack.org we can point out each job name and its relult and when they click it opens a URL . Is a custom version of gerrit used in "review.openstack.org". If so how can i downlaod and install | 05:49 |
| *** baoli has quit IRC | 05:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Neutron: enable a voting py34 gate on master https://review.openstack.org/182439 | 05:54 |
| openstackgerrit | Merged openstack-infra/project-config: Switch on requirements check for Magnum https://review.openstack.org/182337 | 05:55 |
| *** JoshNang has quit IRC | 05:57 | |
| *** JoshNang_ is now known as JoshNang | 05:57 | |
| *** armax has quit IRC | 05:59 | |
| *** dims has joined #openstack-infra | 06:00 | |
| morganfainberg | purp: *waves* just remembered you were on IRC. Annd now back to your regularly scheduled random IRC noise. | 06:02 |
| *** mahito has joined #openstack-infra | 06:03 | |
| *** teran has joined #openstack-infra | 06:03 | |
| *** Alexandra_ has quit IRC | 06:04 | |
| *** Longgeek has joined #openstack-infra | 06:05 | |
| *** dims has quit IRC | 06:06 | |
| *** mahito has quit IRC | 06:07 | |
| *** mahito has joined #openstack-infra | 06:08 | |
| *** zhiwei has quit IRC | 06:09 | |
| *** camunoz has quit IRC | 06:14 | |
| *** yfried has joined #openstack-infra | 06:14 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add a ceph job https://review.openstack.org/176292 | 06:15 |
| *** ildikov has joined #openstack-infra | 06:17 | |
| *** mtanino has quit IRC | 06:17 | |
| *** claudiub has quit IRC | 06:18 | |
| *** Longgeek has quit IRC | 06:22 | |
| *** jcoufal has joined #openstack-infra | 06:24 | |
| *** maishsk_afk has joined #openstack-infra | 06:24 | |
| *** afazekas_ has joined #openstack-infra | 06:25 | |
| *** afazekas has joined #openstack-infra | 06:25 | |
| *** GheRiver1 is now known as GheRivero | 06:25 | |
| *** Sukhdev has quit IRC | 06:29 | |
| *** shardy has joined #openstack-infra | 06:29 | |
| *** sdake has quit IRC | 06:31 | |
| *** otter768 has joined #openstack-infra | 06:34 | |
| *** kevinbenton has quit IRC | 06:36 | |
| *** otter768 has quit IRC | 06:38 | |
| *** woodster_ has quit IRC | 06:50 | |
| *** gema_ is now known as gema | 06:54 | |
| *** mpaolino has joined #openstack-infra | 07:01 | |
| *** HeOS has quit IRC | 07:02 | |
| tchaypo | naggappan: take a look at http://git.openstack.org/cgit | 07:05 |
| tchaypo | naggappan: there are a few gerrit-related repos - http://git.openstack.org/cgit/openstack-infra/gerrit/ and http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/ in particular are probably interesting to you | 07:06 |
| *** sergsh has joined #openstack-infra | 07:06 | |
| *** ZZelle_ has quit IRC | 07:08 | |
| *** ZZelle_ has joined #openstack-infra | 07:08 | |
| *** nfedotov has joined #openstack-infra | 07:09 | |
| GheRivero | morning all | 07:09 |
| *** devvesa has joined #openstack-infra | 07:10 | |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** abregman_ has joined #openstack-infra | 07:22 | |
| *** abregman has quit IRC | 07:24 | |
| *** afazekas_ has quit IRC | 07:25 | |
| *** afazekas has quit IRC | 07:25 | |
| *** mpaolino has quit IRC | 07:25 | |
| *** yamahata has quit IRC | 07:26 | |
| *** afazekas has joined #openstack-infra | 07:27 | |
| *** arxcruz has joined #openstack-infra | 07:28 | |
| *** hichihara has quit IRC | 07:28 | |
| *** jlvillal has joined #openstack-infra | 07:29 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/requirements: Add ironic-discoverd to projects.txt https://review.openstack.org/156270 | 07:30 |
| *** kashyap` is now known as kashyap | 07:31 | |
| *** jistr has joined #openstack-infra | 07:34 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add Keystone service resource methods https://review.openstack.org/177621 | 07:35 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add keystone endpoint resource methods https://review.openstack.org/180328 | 07:36 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Bump integration test timeouts. https://review.openstack.org/182960 | 07:36 |
| *** zhiwei has joined #openstack-infra | 07:37 | |
| *** achanda has quit IRC | 07:38 | |
| *** mahito_ has joined #openstack-infra | 07:41 | |
| Kennan | hi did we upgrade tox version ? | 07:42 |
| Kennan | it seems the latest tox can cause some unit failure | 07:42 |
| *** ajo has joined #openstack-infra | 07:42 | |
| *** jlanoux has joined #openstack-infra | 07:42 | |
| Kennan | the new version tox not same as old | 07:42 |
| *** achanda_ has joined #openstack-infra | 07:42 | |
| Kennan | did we decide to use the new tox ? | 07:42 |
| *** ivar-laz_ has joined #openstack-infra | 07:43 | |
| *** mahito has quit IRC | 07:44 | |
| *** jtomasek has joined #openstack-infra | 07:45 | |
| *** btully has quit IRC | 07:45 | |
| *** ivar-lazzaro has quit IRC | 07:46 | |
| *** geguileo has quit IRC | 07:48 | |
| *** Longgeek has joined #openstack-infra | 07:49 | |
| *** k4n0 has joined #openstack-infra | 07:51 | |
| *** dizquierdo has joined #openstack-infra | 07:57 | |
| *** tnovacik has joined #openstack-infra | 07:57 | |
| lifeless | Kennan: 2.0 is causing issues? The environment isolations tuff perhaps - do you have a link to a failed log or better yet a bug filed? | 07:58 |
| lifeless | Kennan: I don't think an explicit choice was made, we just had an open ended requirement/ | 07:58 |
| SpamapS | yeah I was just looking at where we install it | 07:59 |
| SpamapS | hard to find | 07:59 |
| Kennan | lifeless: for exmaple | 08:00 |
| Kennan | this | 08:00 |
| Kennan | http://logs.openstack.org/89/182589/4/check/gate-magnum-pep8/e226339/console.html | 08:00 |
| Kennan | for pep8 it use tox 2.0.1 | 08:00 |
| *** Longgeek has quit IRC | 08:00 | |
| Kennan | but in http://logs.openstack.org/89/182589/4/check/gate-magnum-python27/5841b0f/console.html | 08:00 |
| *** freerunner has joined #openstack-infra | 08:00 | |
| Kennan | it use tox 1.9.2 | 08:01 |
| Kennan | I don't know how our jenkins env operated | 08:01 |
| Kennan | our local env tox 1.6, it works without issue | 08:01 |
| SpamapS | probably just new images failing, with old images working | 08:01 |
| Kennan | for 1.9.2 the logs in http://logs.openstack.org/89/182589/4/check/gate-magnum-python27/5841b0f/console.html | 08:01 |
| Kennan | it failued for unit | 08:01 |
| *** links has joined #openstack-infra | 08:02 | |
| Kennan | lifeless do you know how our jenkins control tox version ? | 08:02 |
| lifeless | we don't have tox in our global-requirements. An oversight to correct. | 08:03 |
| lifeless | [yes, I know its meta, I don't care :)] | 08:03 |
| lifeless | SpamapS: dvsm-gate probable | 08:03 |
| SpamapS | lifeless: it sits in an odd spot, since tox is used to run allthethings | 08:04 |
| lifeless | SpamapS: sure, but we can document the version we want in the same spot as the rest | 08:04 |
| lifeless | since it participates in the set f things installed that need coinstallability | 08:04 |
| SpamapS | Wouldn't hurt. | 08:04 |
| lifeless | ok, notin dv-gate | 08:04 |
| lifeless | ./lib/tempest: | 08:05 |
| lifeless | in devstack | 08:05 |
| SpamapS | wow | 08:05 |
| SpamapS | really | 08:05 |
| *** freerunner has quit IRC | 08:05 | |
| *** derekh has joined #openstack-infra | 08:05 | |
| *** freerunner has joined #openstack-infra | 08:05 | |
| lifeless | commit 33dc8699e6b774cc39b39e0dba97be41bbb47abe | 08:06 |
| lifeless | Author: Clark Boylan <clark.boylan@gmail.com> | 08:06 |
| lifeless | Date: Tue Aug 19 22:37:10 2014 -0700 | 08:06 |
| lifeless | thats when we went from <1.7 | 08:06 |
| lifeless | to open ended | 08:06 |
| SpamapS | lifeless: it would be worthwhile to have that grep global reqs and just use whatever it finds. | 08:06 |
| lifeless | SpamapS: thats what I'm saying | 08:06 |
| SpamapS | cause yeah, open ended just doesn't work out for us | 08:07 |
| SpamapS | ever | 08:07 |
| lifeless | sure it does | 08:07 |
| lifeless | for 3-4 days at a time | 08:07 |
| SpamapS | and on holidays | 08:07 |
| SpamapS | when nobody is watching or releasing | 08:07 |
| SpamapS | ;) | 08:07 |
| SpamapS | ok snark quota reached, time for bed | 08:08 |
| lifeless | Kennan: SpamapS: what version works? | 08:08 |
| lifeless | like | 08:08 |
| lifeless | 2.0 just came out | 08:08 |
| SpamapS | lifeless: <2.0 | 08:08 |
| lifeless | I presume 2.0 is wat broke? | 08:08 |
| SpamapS | Yeah, I see 1.9 working | 08:08 |
| Kennan | my local use 1.6 worked, not sure newer. (jenkines worked before, just failured today) | 08:09 |
| Kennan | I noticed lifeless | 08:09 |
| lifeless | Kennan: http://logs.openstack.org/89/182589/4/check/gate-magnum-python27/5841b0f/console.html failure doesn't look like its related to tox | 08:09 |
| lifeless | wsme.exc.InvalidInput: Invalid input for field/attribute dns_nameserver. Value: '8.8.1.1'. Value should be IPv4 format | 08:09 |
| StevenK | Haha | 08:09 |
| lifeless | SpamapS: do you have a log of a failure that looks tox related? | 08:09 |
| lifeless | oh and now pypi is down. | 08:10 |
| lifeless | WAT | 08:10 |
| SpamapS | lifeless: http://logs.openstack.org/98/182898/1/check/check-shade-dsvm-functional-neutron/d16eecf/ | 08:11 |
| SpamapS | lifeless: it's not super obvious it is tox's fault.. and we need to work on those errors, but OS_* not passing through is the cause of those failures. | 08:12 |
| SpamapS | and other projects do similar things | 08:12 |
| lifeless | SpamapS: ahha | 08:12 |
| lifeless | SpamapS: ok, so the environment thing. Right | 08:12 |
| SpamapS | and I can reproduce on my test vm | 08:12 |
| Kennan | OS**__ env sometimes can not get, I also noticed such failure in our env | 08:12 |
| Kennan | it is very strange | 08:12 |
| lifeless | Kennan: its in the changelog for tox. | 08:13 |
| SpamapS | so yeah, <2.0 seems prudent | 08:13 |
| lifeless | doing so now | 08:13 |
| SpamapS | and then we probably need to raise the alarm and get onto 2.0 soonish | 08:14 |
| lifeless | I'll send a mail | 08:14 |
| * SpamapS stumbles off to bed | 08:16 | |
| lifeless | mail sent. | 08:16 |
| *** teran has quit IRC | 08:16 | |
| lifeless | https://review.openstack.org/182966 will fix it one hopes | 08:17 |
| *** teran has joined #openstack-infra | 08:18 | |
| *** HeOS has joined #openstack-infra | 08:19 | |
| *** notnownikki has joined #openstack-infra | 08:21 | |
| megm | Hi all. Can someone answer a fairly simple (I hope) git question? | 08:21 |
| *** freerunner has quit IRC | 08:22 | |
| *** spredzy is now known as spredzy|afk | 08:23 | |
| megm | git config --list includes core.editor=vim | 08:23 |
| megm | but when I do git commit, I get error: cannot run vim: No such file or directory | 08:23 |
| megm | error: unable to start editor 'vim' | 08:23 |
| StevenK | megm: What does 'which vim' print out? | 08:25 |
| megm | StevenK: it just returns -- no output. I'm running Ubuntu 14.04, BTW | 08:26 |
| StevenK | megm: Then you don't have vim installed, you should install it by using 'apt-get install vim' as root | 08:27 |
| StevenK | Or Software Centre, etc | 08:27 |
| megm | stevek: really? That's easy enough -- but I run vim all the time so I'm baffled. | 08:28 |
| xnox | only 'vi' is installed by default i believe. | 08:29 |
| xnox | aka vim-minimal, not the full one.... | 08:29 |
| *** btully has joined #openstack-infra | 08:29 | |
| megm | How interesting! But thanks so much -- apt-get worked and git commit now puts me into a vim buffer! | 08:30 |
| lifeless | ed4lyfe | 08:31 |
| *** teran has quit IRC | 08:31 | |
| *** kevinbenton has joined #openstack-infra | 08:32 | |
| xnox | lifeless: troll =) | 08:32 |
| lifeless | xnox: Ahh, you know me | 08:33 |
| xnox | lifeless: being an ex-canonical myself, you are a legend ;-) and have i been told stories about you ;-) | 08:34 |
| *** btully has quit IRC | 08:34 | |
| xnox | lifeless: launchpad team is back picking up people by the way. micro-services & git support are developed et.al. | 08:35 |
| *** teran has joined #openstack-infra | 08:35 | |
| *** otter768 has joined #openstack-infra | 08:35 | |
| lifeless | xnox: yeah, I'm v happy to see that | 08:35 |
| lifeless | I was advocating for microservices before I joined HP | 08:36 |
| lifeless | and git support was a discussion point back then too | 08:36 |
| lifeless | its nice to see it happening | 08:36 |
| xnox | yeah. cjwatson is doing it with wgrant.... and you know shallow clone support available in git =) and with insteadof git setting, one can do | 08:38 |
| xnox | $ git clone lp:project | 08:38 |
| *** btully has joined #openstack-infra | 08:39 | |
| *** otter768 has quit IRC | 08:40 | |
| *** e0ne has quit IRC | 08:40 | |
| *** achanda_ has quit IRC | 08:40 | |
| *** matrohon has joined #openstack-infra | 08:42 | |
| *** btully has quit IRC | 08:44 | |
| *** matrohon has quit IRC | 08:46 | |
| *** SergK_ is now known as SergK | 08:48 | |
| rcarrillocruz | hmm, is docs.openstack.org down? | 08:48 |
| electrical | rcarrillocruz: same for me. | 08:50 |
| megm | docs.openstack.org is working for me | 08:50 |
| electrical | very slow now for me | 08:51 |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Enable Heat services for functional tests https://review.openstack.org/182974 | 08:54 |
| rcarrillocruz | yes, it's crawling... | 08:54 |
| rcarrillocruz | i could open some ironic page from docs.openstack.org, but the front page won't load...at least it's been trying for 3-4 minutes | 08:54 |
| *** achanda has joined #openstack-infra | 08:54 | |
| *** maishsk_afk has quit IRC | 08:57 | |
| *** dtantsur is now known as dtantsur|bbl | 09:04 | |
| *** mkerrin has joined #openstack-infra | 09:05 | |
| *** ssam2 has joined #openstack-infra | 09:05 | |
| *** jhesketh has quit IRC | 09:06 | |
| *** jhesketh has joined #openstack-infra | 09:07 | |
| *** rlandy has joined #openstack-infra | 09:07 | |
| *** jhesketh has quit IRC | 09:08 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 09:09 |
| *** jhesketh has joined #openstack-infra | 09:09 | |
| johnthetubaguy | mordred: sorry, I was sleeping when you pinged me, support is the best option at that point. | 09:11 |
| *** mikal has quit IRC | 09:11 | |
| *** loquacities has quit IRC | 09:12 | |
| *** bnemec has quit IRC | 09:13 | |
| *** bnemec has joined #openstack-infra | 09:13 | |
| *** slagle has quit IRC | 09:14 | |
| *** ivar-laz_ has quit IRC | 09:14 | |
| *** ivar-lazzaro has joined #openstack-infra | 09:15 | |
| *** achanda has quit IRC | 09:15 | |
| *** pelix has joined #openstack-infra | 09:17 | |
| *** ihrachyshka has joined #openstack-infra | 09:20 | |
| *** slagle has joined #openstack-infra | 09:21 | |
| *** achanda has joined #openstack-infra | 09:22 | |
| *** freerunner has joined #openstack-infra | 09:26 | |
| *** ivar-lazzaro has quit IRC | 09:27 | |
| *** ajo has quit IRC | 09:30 | |
| *** mahito_ has quit IRC | 09:30 | |
| *** achanda has quit IRC | 09:31 | |
| openstackgerrit | Richard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x https://review.openstack.org/182988 | 09:32 |
| *** afazekas has quit IRC | 09:32 | |
| *** freerunner has quit IRC | 09:33 | |
| electrical | zaro: as promised: https://review.openstack.org/#/c/182988/ | 09:33 |
| *** freerunner has joined #openstack-infra | 09:34 | |
| *** dims has joined #openstack-infra | 09:36 | |
| *** Longgeek has joined #openstack-infra | 09:36 | |
| *** dims has quit IRC | 09:40 | |
| *** e0ne has joined #openstack-infra | 09:44 | |
| *** e0ne is now known as e0ne_ | 09:44 | |
| *** Longgeek has quit IRC | 09:44 | |
| *** teran has quit IRC | 09:46 | |
| *** mattymo_ is now known as mattymo | 09:49 | |
| *** e0ne_ is now known as e0ne | 09:52 | |
| *** derekh has quit IRC | 09:55 | |
| openstackgerrit | Richard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x https://review.openstack.org/182988 | 09:57 |
| *** afazekas has joined #openstack-infra | 09:58 | |
| *** ivar-lazzaro has joined #openstack-infra | 09:58 | |
| *** ivar-lazzaro has quit IRC | 10:03 | |
| openstackgerrit | Luis Cañas-Díaz proposed openstack-infra/activity-board: Update JS library and company panel to add top people by company https://review.openstack.org/182995 | 10:06 |
| *** dims has joined #openstack-infra | 10:07 | |
| *** weshay has quit IRC | 10:07 | |
| *** cdent has joined #openstack-infra | 10:08 | |
| *** weshay has joined #openstack-infra | 10:09 | |
| *** amotoki has quit IRC | 10:10 | |
| *** dguerri is now known as _dguerri | 10:11 | |
| *** enikanorov2 has quit IRC | 10:15 | |
| *** _dguerri is now known as dguerri | 10:15 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Switch on docs and publish jobs for magnum https://review.openstack.org/182998 | 10:16 |
| *** ajo has joined #openstack-infra | 10:16 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Switch on docs and publish jobs for magnum https://review.openstack.org/182998 | 10:22 |
| *** kevinbenton has quit IRC | 10:25 | |
| *** kevinbenton has joined #openstack-infra | 10:26 | |
| *** sdake has joined #openstack-infra | 10:34 | |
| *** otter768 has joined #openstack-infra | 10:36 | |
| *** teran has joined #openstack-infra | 10:37 | |
| *** ppai has quit IRC | 10:39 | |
| *** teran_ has joined #openstack-infra | 10:40 | |
| *** otter768 has quit IRC | 10:40 | |
| *** yfried is now known as yfried|afk | 10:42 | |
| *** zhiwei has quit IRC | 10:43 | |
| *** teran has quit IRC | 10:43 | |
| *** ihrachyshka_ has joined #openstack-infra | 10:46 | |
| *** dguerri is now known as _dguerri | 10:47 | |
| *** ihrachyshka has quit IRC | 10:49 | |
| electrical | fungi: pelix yolanda not sure if any of you can have a look at https://review.openstack.org/#/c/182988/ test is failing of a part i didn't touch so it must have been failing before ? | 10:49 |
| *** abregman_ has quit IRC | 10:50 | |
| pelix | electrical: you need to remove the use of decode from the tests | 10:50 |
| *** yamamoto has quit IRC | 10:51 | |
| pelix | In fixing the code you've broken the tests, but then the tests were wrong anyway, in that they shouldn't have been using decode | 10:51 |
| pelix | that should have been done by the code they were testing | 10:51 |
| electrical | Ah okay. i only see 1 event of using decode in the tests | 10:52 |
| sdague | lifeless: there isn't much meat on your email or the review and lots of stuff is passing, so "gate wedged" needs to be more specific as to what jobs seem to be impacted | 10:52 |
| *** ppai has joined #openstack-infra | 10:52 | |
| electrical | pelix: will remove it and re-submit | 10:53 |
| electrical | thanks for the quick check :-) | 10:53 |
| pelix | electrical: np | 10:56 |
| openstackgerrit | Richard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x https://review.openstack.org/182988 | 10:57 |
| electrical | pelix: ^^ that should solve it then i hope :-) | 10:57 |
| *** jamielennox is now known as jamielennox|away | 10:58 | |
| *** ppai has quit IRC | 10:58 | |
| *** dizquierdo has quit IRC | 10:58 | |
| *** ivar-lazzaro has joined #openstack-infra | 10:59 | |
| sambetts | Hey, is there anyway to clean up the CI comments on a patch? A couple of weeks ago, A10 networks CI went baserk and left hundreds of comments on a couple of my patchs which makes loading the review page for those patches really slow, I think because of the JS processing the comments into that nice box, any ideas how to fix this or is it just going to have to be like that until that patch merges? | 10:59 |
| *** sdake_ has joined #openstack-infra | 10:59 | |
| *** abregman_ has joined #openstack-infra | 11:02 | |
| *** sdake has quit IRC | 11:02 | |
| *** dtantsur|bbl is now known as dtantsur | 11:03 | |
| *** ivar-lazzaro has quit IRC | 11:04 | |
| pelix | electrical: think it might have been my mistake, it's the encode calls that need to be looked at | 11:04 |
| *** mwagner_lap has joined #openstack-infra | 11:05 | |
| *** weshay has quit IRC | 11:06 | |
| pelix | electrical: ah, no I spotted the problem, there were additional decode calls being made in the python jenkins, in decoding all responses before returning them, needed to remove extraneous decodes from elsewhere. | 11:06 |
| electrical | pelix: ahhh okay. makes sense since we have now a single place doing it | 11:06 |
| electrical | pelix: removing it from the test was still a good move or do i need to place it back? | 11:07 |
| sdague | sambetts: no, deleting comments isn't really a thing | 11:08 |
| pelix | I think that one needs to be put back | 11:08 |
| electrical | pelix: okay | 11:08 |
| pelix | since it's inspecting the attribute directly rather than the response | 11:09 |
| electrical | pelix: i'm having quite some problems running the tests my self. using pyenv to select the python version. running tox fails continuously for me locally | 11:09 |
| electrical | even running python setup.py testr --slowest doesn't work | 11:10 |
| pelix | electrical: I'm taking a look at it now as I have 3.4 and 2.7 installed via my system ;) | 11:10 |
| sambetts | sdague: If you want to see what I mean check out this patch https://review.openstack.org/#/c/161381/ toggle CI if you have to, to see all the comments and then scroll down, its insane and shame I can't get it cleaned up | 11:10 |
| electrical | pelix: ah okay thanks :-) | 11:11 |
| sdague | sambetts: right, I'm aware what happens, but deleting comments isn't really a thing as far as I know | 11:11 |
| *** derekh has joined #openstack-infra | 11:11 | |
| *** ppai has joined #openstack-infra | 11:11 | |
| sambetts | sdague: :( ah well | 11:12 |
| *** e0ne is now known as e0ne_ | 11:12 | |
| pelix | the places that need to be changed are the removal of encode of the json.dumps(data) on the test_jenkins_open test, add back in the decode on the j.auth, and removal of the decode in the maybe_add_crumb | 11:12 |
| *** pelix has quit IRC | 11:13 | |
| *** panda has quit IRC | 11:13 | |
| *** pelix has joined #openstack-infra | 11:14 | |
| *** deepakcs has quit IRC | 11:14 | |
| *** panda has joined #openstack-infra | 11:14 | |
| *** loquacities has joined #openstack-infra | 11:15 | |
| *** e0ne_ is now known as e0ne | 11:19 | |
| *** ihrachyshka_ has quit IRC | 11:20 | |
| openstackgerrit | Richard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x https://review.openstack.org/182988 | 11:21 |
| electrical | pelix: ^^ that should be the final fix i hope :-) | 11:23 |
| *** yfried|afk is now known as yfried | 11:24 | |
| *** banix has joined #openstack-infra | 11:25 | |
| *** woodster_ has joined #openstack-infra | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:26 | |
| *** banix has quit IRC | 11:27 | |
| *** banix has joined #openstack-infra | 11:27 | |
| electrical | pelix: an other error now :-( | 11:28 |
| *** doug-fish has joined #openstack-infra | 11:28 | |
| *** Longgeek has joined #openstack-infra | 11:29 | |
| *** ihrachyshka_ has joined #openstack-infra | 11:32 | |
| electrical | pelix: managed to get test runs to work. fixed the issue. pushing new patch | 11:33 |
| openstackgerrit | Richard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x https://review.openstack.org/182988 | 11:33 |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** banix has quit IRC | 11:35 | |
| electrical | pelix: finally a +1 from jenkins :D | 11:38 |
| *** lil_cain has joined #openstack-infra | 11:39 | |
| *** fhubik has joined #openstack-infra | 11:39 | |
| lil_cain | Hey - I have some jenkins failures - is there an easy way to tell what platform/pypy version jenkins is running the tests on? | 11:41 |
| *** weshay has joined #openstack-infra | 11:44 | |
| *** chlong has joined #openstack-infra | 11:48 | |
| *** soren_ has joined #openstack-infra | 11:49 | |
| *** baoli has joined #openstack-infra | 11:50 | |
| *** freerunner has quit IRC | 11:52 | |
| *** luqas__ has quit IRC | 11:52 | |
| *** pc_m has joined #openstack-infra | 11:52 | |
| *** ssam2 has left #openstack-infra | 11:55 | |
| *** _dguerri is now known as dguerri | 11:58 | |
| *** mwagner_lap has quit IRC | 11:58 | |
| *** devvesa has quit IRC | 11:59 | |
| *** ivar-lazzaro has joined #openstack-infra | 11:59 | |
| *** freerunner has joined #openstack-infra | 12:00 | |
| openstackgerrit | Victor Seva proposed openstack-infra/jenkins-job-builder: Allow use of aliases defined previously inside included files https://review.openstack.org/157735 | 12:01 |
| *** Longgeek has quit IRC | 12:01 | |
| openstackgerrit | Ruslan Kamaldinov proposed openstack-infra/project-config: Mark apps-catalog py34 job as non-voting https://review.openstack.org/183019 | 12:01 |
| *** dizquierdo has joined #openstack-infra | 12:01 | |
| *** ivar-lazzaro has quit IRC | 12:04 | |
| linuxmaniac | zaro: Thanks for the review. I've reduced the xml produced as you asked. | 12:05 |
| *** e0ne is now known as e0ne_ | 12:05 | |
| *** e0ne_ is now known as e0ne | 12:06 | |
| *** zhiwei has joined #openstack-infra | 12:06 | |
| *** ihrachyshka_ has quit IRC | 12:08 | |
| *** gordc has joined #openstack-infra | 12:10 | |
| *** devvesa has joined #openstack-infra | 12:11 | |
| *** claudiub has joined #openstack-infra | 12:13 | |
| *** claudiub|2 has joined #openstack-infra | 12:14 | |
| *** dprince has joined #openstack-infra | 12:16 | |
| *** claudiub|3 has joined #openstack-infra | 12:17 | |
| *** claudiub_ has joined #openstack-infra | 12:18 | |
| *** claudiub has quit IRC | 12:18 | |
| *** claudiub has joined #openstack-infra | 12:18 | |
| *** rfolco has joined #openstack-infra | 12:18 | |
| *** claudiub|2 has quit IRC | 12:19 | |
| *** claudiub|2 has joined #openstack-infra | 12:20 | |
| *** claudiub|3 has quit IRC | 12:21 | |
| *** claudiub|3 has joined #openstack-infra | 12:22 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** coolsvap is now known as coolsvap|afk | 12:22 | |
| *** claudiub|4 has joined #openstack-infra | 12:23 | |
| *** claudiub has quit IRC | 12:23 | |
| *** claudiub has joined #openstack-infra | 12:24 | |
| *** ihrachyshka_ has joined #openstack-infra | 12:24 | |
| *** kgiusti has joined #openstack-infra | 12:24 | |
| *** claudiub|2 has quit IRC | 12:25 | |
| *** BharatK has quit IRC | 12:26 | |
| *** claudiub|3 has quit IRC | 12:26 | |
| *** claudiub|4 has quit IRC | 12:27 | |
| *** bhunter71 has joined #openstack-infra | 12:29 | |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| *** markvoelker has quit IRC | 12:31 | |
| *** markvoelker has joined #openstack-infra | 12:31 | |
| *** claudiub|2 has joined #openstack-infra | 12:32 | |
| *** zz_dimtruck is now known as dimtruck | 12:33 | |
| *** claudiub_ has quit IRC | 12:34 | |
| *** kadm__ has joined #openstack-infra | 12:34 | |
| *** claudiub has quit IRC | 12:34 | |
| *** BharatK has joined #openstack-infra | 12:35 | |
| *** claudiub|2 has quit IRC | 12:36 | |
| *** kadm__ has quit IRC | 12:36 | |
| *** kadm__ has joined #openstack-infra | 12:37 | |
| *** otter768 has joined #openstack-infra | 12:37 | |
| electrical | pelix: could i pick your brains sometime on how i should template the jenkins yaml files for jjb in the best way? I have allot of jobs for the same 'product' ( puppet module ) and each product has the same set of jobs. there are also 3 jobs that are dependent on each other ( 1 master job with 2 jobs that get triggered of it | 12:38 |
| *** erlon has joined #openstack-infra | 12:38 | |
| *** stevemar has joined #openstack-infra | 12:38 | |
| *** yfried is now known as yfried|afk | 12:40 | |
| *** yfried|afk is now known as yfried | 12:41 | |
| *** otter768 has quit IRC | 12:41 | |
| *** bknudson has quit IRC | 12:42 | |
| *** sdake_ has quit IRC | 12:43 | |
| *** kadm__ has quit IRC | 12:44 | |
| *** sdake has joined #openstack-infra | 12:44 | |
| *** ihrachyshka_ has quit IRC | 12:46 | |
| *** mwagner_lap has joined #openstack-infra | 12:46 | |
| *** bswartz has quit IRC | 12:46 | |
| *** aysyanne has joined #openstack-infra | 12:48 | |
| *** e0ne is now known as e0ne_ | 12:51 | |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Add glusterfs/centos7 experimental job https://review.openstack.org/175626 | 12:52 |
| *** abregman_ has quit IRC | 12:52 | |
| *** e0ne_ is now known as e0ne | 12:53 | |
| *** yfried has quit IRC | 12:56 | |
| *** gordc has quit IRC | 12:56 | |
| *** kadm__ has joined #openstack-infra | 12:56 | |
| *** ihrachyshka_ has joined #openstack-infra | 12:57 | |
| *** radez_g0n3 is now known as radez | 12:59 | |
| *** dguerri is now known as _dguerri | 12:59 | |
| *** sarob has joined #openstack-infra | 12:59 | |
| *** ivar-lazzaro has joined #openstack-infra | 13:00 | |
| *** _dguerri is now known as dguerri | 13:01 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 13:02 | |
| *** claudiub has joined #openstack-infra | 13:03 | |
| *** heyongli has joined #openstack-infra | 13:03 | |
| *** kadm__ has quit IRC | 13:03 | |
| heyongli | anteaya, ping | 13:04 |
| *** ppai has quit IRC | 13:05 | |
| *** kadm__ has joined #openstack-infra | 13:05 | |
| *** ivar-lazzaro has quit IRC | 13:05 | |
| *** abregman_ has joined #openstack-infra | 13:05 | |
| *** afazekas has quit IRC | 13:05 | |
| *** derekh_ has joined #openstack-infra | 13:07 | |
| *** kadm__ has quit IRC | 13:07 | |
| *** mestery has joined #openstack-infra | 13:08 | |
| *** afazekas has joined #openstack-infra | 13:08 | |
| electrical | fungi: morning :-) | 13:08 |
| *** ociuhandu has joined #openstack-infra | 13:09 | |
| fungi | lil_cain: the pypy 2.2.1+dfsg-1ubuntu0.2 package provided in the ubuntu 14.04 lts amd64 updates repo (trusty-updates) | 13:09 |
| fungi | electrical: good afternoon(?) to you! | 13:09 |
| lil_cain | fungi: Deadly, thanks | 13:09 |
| *** annegentle has joined #openstack-infra | 13:10 | |
| electrical | fungi: hehe yeah. 2 pm | 13:10 |
| *** derekh has quit IRC | 13:10 | |
| *** dkranz has joined #openstack-infra | 13:11 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch on docs and publish jobs for magnum https://review.openstack.org/182998 | 13:12 |
| *** devvesa has quit IRC | 13:14 | |
| *** alaski has quit IRC | 13:14 | |
| *** Longgeek has joined #openstack-infra | 13:15 | |
| *** miqui has joined #openstack-infra | 13:16 | |
| *** alaski has joined #openstack-infra | 13:16 | |
| *** ppai has joined #openstack-infra | 13:16 | |
| *** bswartz has joined #openstack-infra | 13:18 | |
| *** peristeri has joined #openstack-infra | 13:18 | |
| sdague | fungi: has rax scheduled the world reboot yet? | 13:21 |
| fungi | sdague: we did it yesterday | 13:21 |
| sdague | oh, gotcha | 13:22 |
| sdague | cool :) | 13:22 |
| fungi | they sent notifications to everyone to perform hard reboots of all hvpvm servers within 24 hours or they would do it for us | 13:22 |
| *** ddieterly has joined #openstack-infra | 13:22 | |
| *** soren_ has quit IRC | 13:22 | |
| fungi | sp we engaged in that pleasant scramble yesterday evening | 13:22 |
| *** ppai has quit IRC | 13:23 | |
| fungi | though i'm marginally worried... for my personal servers i did the same, and rackspace keeps opening new tickets reminding me to reboot them even though i responded in the originals that i had done so and then closed them | 13:23 |
| *** dboik_ has quit IRC | 13:23 | |
| electrical | fungi: I'm working on a new module for jjb. logstash reporter. almost done i think | 13:23 |
| fungi | electrical: neat. we don't really use any jenkins plugins except for the couple which we have as glue into zuul and nodepool (gearman and zeromq plugins respectively), but it's nice to see a healthy ecosystem plugin support layer growing in jjb | 13:24 |
| *** changbl has quit IRC | 13:25 | |
| annegentle | fungi: yuck, I'll chase that. | 13:25 |
| electrical | fungi: hehe yeah indeed. I'll most likely only build the ones I'm using :-) | 13:25 |
| *** bhunter71 has quit IRC | 13:25 | |
| fungi | annegentle: the rackspace tickets? yeah makes me wonder if those are just repeats from their original list of affected servers, or if they really don't know whether anyone has rebooted anything | 13:26 |
| annegentle | fungi: right. I have more emails from tickets in my inbox this morning. | 13:26 |
| mtreinish | fungi: heh, I closed my rax account last week since it won't be free anymore | 13:26 |
| fungi | and are just going to reboot them all again even though we already did | 13:26 |
| mtreinish | so don't have to worry about reboots :) | 13:27 |
| annegentle | mtreinish: heh, this one's for all the clouds tho | 13:27 |
| *** heyongli has quit IRC | 13:27 | |
| fungi | annegentle: righth, for my personal servers alone i think i got 20 rackspace ticket notification e-mails in the last 24 hours (as i have instances in iad and syd and they treated them separately) | 13:27 |
| annegentle | fungi: I don't know much about the automation | 13:27 |
| annegentle | fungi: right. Our customers hate all the ticket flood. | 13:27 |
| *** freerunner has quit IRC | 13:28 | |
| mtreinish | annegentle: yeah, I saw the cve, but rax was the only cloud I was really using. I have my hp cloud account but I don't really use it for much | 13:28 |
| electrical | fungi: jenkins_jobs.errors.JenkinsJobsException: Unknown entry point or macro 'logstash' for component type: 'reporter'. any idea? i created the logstash def in reporters but test fails. | 13:28 |
| fungi | electrical: is it in the entrypoints list? have you looked at a recent plugin addition commit for ideas? | 13:30 |
| electrical | fungi: haven't checked to be very honest. my fault. | 13:31 |
| *** prad has quit IRC | 13:31 | |
| *** prad has joined #openstack-infra | 13:31 | |
| annegentle | mtreinish: workin' for cloud creds :) | 13:31 |
| electrical | Ah, i see what i'm missing | 13:32 |
| fungi | electrical: something like https://review.openstack.org/171048 might be useful to review | 13:32 |
| fungi | electrical: so wondering if it's in setup.cfg | 13:33 |
| electrical | fungi: it wasn't no. | 13:33 |
| electrical | that error is gone now. only got a discrepency between the yaml and xml file | 13:33 |
| *** gordc has joined #openstack-infra | 13:33 | |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras and env markers https://review.openstack.org/181795 | 13:34 |
| *** jgrimm is now known as zz_jgrimm | 13:35 | |
| linuxmaniac | zaro: Thanks! :-) | 13:35 |
| mordred | fungi: so - fwiw, tox 2.0 broke shade functional tests | 13:35 |
| *** enikanorov2 has joined #openstack-infra | 13:36 | |
| fungi | mordred: so i heard. started filtering the calling env | 13:36 |
| mordred | yup | 13:36 |
| mordred | it's fine - I want that to be consuming a clouds.yaml input file _anyway_ | 13:36 |
| *** dustins has joined #openstack-infra | 13:36 | |
| mordred | but I was ignoring doing the work for that since the env var hack was working | 13:36 |
| fungi | long-term i think i like that change, since it's nice to be explicit about what parts of tox's calling environment should be seen in the test envs | 13:37 |
| mordred | yah | 13:37 |
| mtreinish | mordred: fwiw, I think you should be able to just do passenv = * and everything will work again | 13:37 |
| mordred | Shrews: ^^ | 13:37 |
| mtreinish | annegentle: heh, yeah something like that :) | 13:37 |
| mordred | Shrews: that might be an answer for the short term | 13:37 |
| fungi | yep, that's what keystone ended up doing | 13:37 |
| mtreinish | mordred: http://tox.readthedocs.org/en/latest/config.html#confval-passenv=SPACE-SEPARATED-GLOBNAMES | 13:37 |
| mordred | fungi: greghaynes had the idea yesterday that was similar to one I'd also had - which is to add a command line program to os-client-config that will take OS_ envvars from the environment and write them into a clouds.yaml file for you | 13:38 |
| mordred | fungi: so you could say "source foo.env ; os-client-config add-cloud devstack" | 13:38 |
| mtreinish | I've got the tempest review up, although I figured I'd try to limit it to the vars that are being used: https://review.openstack.org/183035 | 13:38 |
| *** mriedem_away is now known as mriedem | 13:38 | |
| Shrews | mordred: oh, that's a neat idea | 13:38 |
| mordred | Shrews: yah. should make it easy for people to consume new cloud accounts | 13:38 |
| fungi | it's also worth noting that tox's current documentation still mentions the {envbindir} parameter as supported, but 2.0 seems to have broken/dropped it. bknudson opened a bug report for holger about that | 13:38 |
| mordred | btw - ec2 and google each have a command line tool you use for setting your local credentials | 13:39 |
| mordred | Shrews: we shoudl also honestly make it able to take an env vars file on the command line too - because needing to source them is a little lame | 13:40 |
| fungi | that might be useful is os-cloud-config, maybe presented by openstackclient when it's missing credentials? | 13:40 |
| fungi | er, s/is/in/ | 13:40 |
| mordred | yah | 13:40 |
| *** dguerri is now known as _dguerri | 13:40 | |
| mordred | fungi: there it could be "openstack cloud add" or something | 13:40 |
| mordred | dtroyer: ^^ | 13:40 |
| fungi | <clippy>it looks like you're trying to openstack</clippy> | 13:40 |
| * Shrews stabs fungi's clippy | 13:40 | |
| mordred | Shrews: we should also convince david-lyle to add "download a clouds.yaml snippet" to horizon | 13:41 |
| *** bknudson has joined #openstack-infra | 13:41 | |
| mordred | Shrews: so that one of the things your horizon can do for you is give you a clouds.yaml for your cloud account - and os-client-config should then be able to merge that in to your current clouds.yaml locally | 13:41 |
| fungi | that's a helpful entry point for newbie users too, agreed | 13:42 |
| *** pabelanger has joined #openstack-infra | 13:42 | |
| *** pabelanger has joined #openstack-infra | 13:42 | |
| dtroyer | adding that to osc should be simple… | 13:42 |
| dtroyer | I'd rather see it there than in a library | 13:42 |
| *** claudiub has quit IRC | 13:42 | |
| *** arielb has joined #openstack-infra | 13:43 | |
| *** arielb has quit IRC | 13:43 | |
| *** arielb has joined #openstack-infra | 13:43 | |
| Shrews | mordred: so not just env vars merge, but also separate clouds.yaml merges? | 13:43 |
| mordred | yah | 13:43 |
| fungi | whichever. the only reason i was thinking of putting the guts in a lib is so that more apps than openstackclient could reuse the same config writing backend | 13:43 |
| *** _dguerri is now known as dguerri | 13:43 | |
| mordred | "I have this cloud defined in this clouds.yaml, please add it to my others" | 13:43 |
| dtroyer | guts, sure, not cli | 13:43 |
| mordred | ++ | 13:43 |
| pabelanger | morning | 13:43 |
| fungi | agreed. the cli for it goes in the client. full stop | 13:43 |
| mordred | guts -> o-c-c, cli -> osc | 13:43 |
| *** dboik has joined #openstack-infra | 13:46 | |
| *** e0ne is now known as e0ne_ | 13:47 | |
| *** Longgeek has quit IRC | 13:48 | |
| sdague | mordred: so I approached the python-novaclient functional tests differently, and just put stuff in a file | 13:48 |
| sdague | https://review.openstack.org/#/c/183007/ | 13:48 |
| sdague | mtreinish: so... the tempest tests are running without that, we just loosing some safety net? | 13:49 |
| mtreinish | sdague: yeah, the gate uses defaults anyway or just overrides things in the tox.ini | 13:50 |
| mtreinish | it's really for people using the options outside of the gate | 13:50 |
| openstackgerrit | Richard Pijnenburg proposed openstack-infra/jenkins-job-builder: Add logstash reporter https://review.openstack.org/183048 | 13:50 |
| mordred | sdague: so ... just for sake of argument | 13:50 |
| sdague | so... for outside of the gate, that seems like those should go in the config instead | 13:50 |
| *** freerunner has joined #openstack-infra | 13:51 | |
| mordred | sdague: why not use o-c-c? | 13:51 |
| mordred | and/or would you accept such a patch? | 13:51 |
| electrical | fungi: https://review.openstack.org/#/c/183048/ would love your feedback on it. | 13:51 |
| sdague | mordred: mostly because I'm not sure exactly what that would look like | 13:51 |
| mtreinish | sdague: it's mostly for the telling tempest where the config file is, the only way we could pass that in was via an env var | 13:51 |
| sdague | mtreinish: right, that one, fine | 13:51 |
| sdague | until there is a real cli | 13:52 |
| mtreinish | yep | 13:52 |
| sdague | but the rest? | 13:52 |
| mordred | sdague: ok. how about I make you a patch and you will like it or not | 13:52 |
| sdague | mordred: that sounds completely reasonble to me | 13:52 |
| mordred | \o/ | 13:52 |
| mtreinish | sdague: the rest are triggers for fixtures, I mean I guess we can move that into the tempest config | 13:52 |
| mordred | sdague: do you mind if I hijack that patch? or would you rather I do a followup one? | 13:53 |
| mtreinish | oh, and 1 is to set the discovery path (but tox should own that anyway, so we can run unit tests) | 13:53 |
| sdague | mordred: follow up please | 13:53 |
| sdague | so that we can unwedge for now | 13:53 |
| mordred | sdague: ok | 13:54 |
| *** freerunner has quit IRC | 13:54 | |
| sdague | and contemplate better | 13:54 |
| sdague | if you stack it on top, that would be cool | 13:54 |
| mordred | I shall do that | 13:54 |
| *** freerunner has joined #openstack-infra | 13:54 | |
| mordred | I'm also going to write the devstack patch I keep talking about about outputting a clouds.yaml file in the first place | 13:54 |
| *** freerunner has quit IRC | 13:54 | |
| sdague | mordred: cool | 13:54 |
| mordred | now that we have more than one thing that know how to read them | 13:54 |
| mordred | actually, I'll do that first | 13:55 |
| *** ddieterl_ has joined #openstack-infra | 13:55 | |
| sdague | anyone know if we're actually tracking a bug here for the tox thing? | 13:55 |
| mordred | and then make the novaclient change make use of it | 13:55 |
| *** mestery has quit IRC | 13:55 | |
| mtreinish | sdague: I haven't seen one yet | 13:55 |
| *** pabelanger has quit IRC | 13:55 | |
| mtreinish | but I didn't look too hard | 13:55 |
| sdague | hey, lets do that then | 13:56 |
| mordred | sdague: I don't suppose you'd entertain a patch to s/OS_TENANT_NAME/OS_PROJECT_NAME/ | 13:57 |
| *** pabelanger has joined #openstack-infra | 13:57 | |
| *** ddieterly has quit IRC | 13:57 | |
| sdague | mordred: you can overload it | 13:57 |
| mordred | I know | 13:57 |
| mtreinish | it'll always be tenant to me :) | 13:57 |
| sdague | but we'll need a deprecation cycle | 13:57 |
| mordred | I was just reading a comment that says tenant is the new "right" thing | 13:58 |
| sdague | because people are going to have stuff hardcoded | 13:58 |
| sdague | and figure out how to warn them | 13:58 |
| mordred | and it was the comment I was more concerned with, since that's no longer accurate | 13:58 |
| sdague | comment changes are fine | 13:58 |
| mordred | and if they're learning by reading comments, we're perpetuating things | 13:58 |
| mordred | cool | 13:58 |
| fungi | i don't know of any one specific bug. i expect that any bugs open about this are particular to the different ways in which tox behavior has changed to impact various repos/jobs | 13:58 |
| mordred | I mean, I probably won't do that today | 13:58 |
| sdague | I'll file one | 13:58 |
| *** mtanino has joined #openstack-infra | 14:00 | |
| *** baoli has quit IRC | 14:00 | |
| sdague | https://bugs.launchpad.net/openstack-gate/+bug/1455102 | 14:00 |
| openstack | Launchpad bug 1455102 in OpenStack-Gate "some test jobs broken by tox 2.0 not passing env variables" [Undecided,Confirmed] | 14:00 |
| *** baoli has joined #openstack-infra | 14:01 | |
| *** ivar-lazzaro has joined #openstack-infra | 14:01 | |
| *** dboik_ has joined #openstack-infra | 14:02 | |
| openstackgerrit | Merged openstack-infra/system-config: Move afs servers to using o_p::template https://review.openstack.org/180783 | 14:02 |
| *** tiswanso has joined #openstack-infra | 14:02 | |
| *** btully has joined #openstack-infra | 14:02 | |
| *** gokrokve has joined #openstack-infra | 14:03 | |
| fungi | nibalizer: that ^ merged. i'll try to make sure it's puppeting successfully | 14:03 |
| *** dkranz has quit IRC | 14:04 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:05 | |
| *** dboik has quit IRC | 14:06 | |
| *** e0ne_ is now known as e0ne | 14:06 | |
| *** ivar-lazzaro has quit IRC | 14:06 | |
| *** rossella_s has quit IRC | 14:07 | |
| *** rossella_s has joined #openstack-infra | 14:07 | |
| mordred | fungi: woot! | 14:08 |
| *** sputnik13 has joined #openstack-infra | 14:08 | |
| mordred | sdague: ok. so - for devstack->clouds.yaml - it seems like I should either make a new script that gets called aroudn the same time as create_userrrc - or I should make create_userrc do it since it already knows the things | 14:10 |
| mordred | sdague, dtroyer: ^^ do you have preference? | 14:10 |
| *** mattfarina has joined #openstack-infra | 14:10 | |
| *** cody-somerville has quit IRC | 14:10 | |
| *** ildikov has quit IRC | 14:10 | |
| mordred | although it might just want to be a different thing, because it wants to know some of the things that go into .stackenv too | 14:11 |
| *** sarob has quit IRC | 14:11 | |
| mordred | sdague, dtroyer: nevermind - let me play a while longer and come back and ask questions | 14:11 |
| dtroyer | I was thinking it would only have one cloud in it… were you thinking a distinct entry for each user account? | 14:12 |
| *** markvoelker has quit IRC | 14:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-nodepool: Consume puppet-diskimage_builder https://review.openstack.org/167610 | 14:14 |
| mordred | dtroyer: unsure? | 14:14 |
| mordred | dtroyer: I was originally thinking one cloud too - but the user account thing is an interesting one | 14:14 |
| fungi | i am interested in support for multiple user accounts in one tenant, fwiw. that is actually my personal workflow for my systems | 14:17 |
| fungi | i have a low-privilege user with read-only access to the api and a high-privilege one i use for creating/deleting servers, et cetera | 14:17 |
| mtreinish | mordred, dtroyer: something like this: http://git.openstack.org/cgit/openstack/tempest/tree/etc/accounts.yaml.sample | 14:17 |
| fungi | so i switch between credentials depending on what calls i'm initiating, as a safeguard | 14:18 |
| mordred | fungi: yup. that's kinda what o-c-c is there for - although each one would be a "cloud" | 14:18 |
| mordred | but yeah - named sets of cloud credentials | 14:18 |
| fungi | oh, got it | 14:18 |
| mordred | mtreinish: I should port that to use o-c-c too ... | 14:18 |
| mordred | mtreinish: just kidding - that's a bit different usecase | 14:18 |
| dtroyer | btw, this is also why I want to keep the ability to load a cloud and override individual things like USERNAME from the env | 14:19 |
| mordred | given the roles | 14:19 |
| fungi | yeah, that serves my purposes. i should get around to using that. at the moment i just have scripts sourcing the appropriate credentials, prompting me to decrypt the corresponding password and passing them into openstackclient | 14:19 |
| mtreinish | mordred: heh, yeah jamielennox|away brought that up before. They're similar but different enough | 14:19 |
| mordred | dtroyer: yeah - loading a cloud and overriding from the env gets _REALLY_ confusing for the common case though | 14:19 |
| mtreinish | mordred: yeah roles, and optionally specifying a network | 14:19 |
| *** unicell1 has joined #openstack-infra | 14:20 | |
| *** unicell has quit IRC | 14:20 | |
| *** dkranz has joined #openstack-infra | 14:20 | |
| *** claudiub has joined #openstack-infra | 14:21 | |
| *** jlanoux_ has joined #openstack-infra | 14:26 | |
| *** ildikov has joined #openstack-infra | 14:26 | |
| *** fhubik has quit IRC | 14:26 | |
| *** akshai has joined #openstack-infra | 14:26 | |
| *** k4n0 has quit IRC | 14:26 | |
| *** fhubik has joined #openstack-infra | 14:26 | |
| *** mkerrin has quit IRC | 14:27 | |
| *** aswadr has joined #openstack-infra | 14:27 | |
| *** jlanoux has quit IRC | 14:28 | |
| openstackgerrit | Merged openstack-infra/release-tools: Add list_repos_by_project.py https://review.openstack.org/179244 | 14:29 |
| openstackgerrit | Merged openstack-infra/release-tools: Add list_unreleased_changes.sh https://review.openstack.org/179245 | 14:29 |
| openstackgerrit | Merged openstack-infra/release-tools: Add list_oslo_unreleased_changes.sh https://review.openstack.org/179396 | 14:29 |
| openstackgerrit | Merged openstack-infra/release-tools: Move get_last_tag to functions https://review.openstack.org/179444 | 14:29 |
| openstackgerrit | Merged openstack-infra/release-tools: Print release notes as part of releasing the library https://review.openstack.org/179445 | 14:29 |
| *** eharney has joined #openstack-infra | 14:30 | |
| wznoinsk | hi, is anyone here? | 14:31 |
| *** achanda has joined #openstack-infra | 14:33 | |
| mordred | wznoinsk: there are over 300 people here ;) | 14:33 |
| wznoinsk | I'm wondering is there an issue in gerrit, i.e.: https://review.openstack.org/#/c/183027/ doesn't have Jenkins and many other CI's in the CI isummary table at the top | 14:34 |
| *** smarcet has joined #openstack-infra | 14:34 | |
| fungi | my irc client claims 429, but some of those are bots | 14:34 |
| *** claudiub has quit IRC | 14:34 | |
| openstackgerrit | Merged openstack-infra/release-tools: Add script to update .gitreview files https://review.openstack.org/174594 | 14:34 |
| openstackgerrit | Merged openstack-infra/release-tools: Submit .gitreview to the right branch https://review.openstack.org/182818 | 14:35 |
| fungi | wznoinsk: have you checked http://status.openstack.org/zuul/ to see if it's in the process of being tested still? | 14:35 |
| wznoinsk | sorry, bad example, give me a sec | 14:36 |
| fungi | wznoinsk: looks like it's been in the check pipeline for 19 minutes, seems to be passing all jobs which have completed so far, and has an estimated 35 minutes to report | 14:36 |
| *** toabctl has joined #openstack-infra | 14:37 | |
| *** tnovacik has quit IRC | 14:37 | |
| *** fhubik has quit IRC | 14:38 | |
| *** Shrews has quit IRC | 14:38 | |
| *** fhubik has joined #openstack-infra | 14:39 | |
| *** gilliard has quit IRC | 14:39 | |
| *** markvoelker has joined #openstack-infra | 14:40 | |
| *** mordred has quit IRC | 14:40 | |
| *** nelsnelson has joined #openstack-infra | 14:40 | |
| wznoinsk | fungi, yes, you're right, didn't realize there was another patchset already, no issue then | 14:41 |
| *** gilliard has joined #openstack-infra | 14:41 | |
| *** fhubik is now known as fhubik_afk | 14:41 | |
| *** mordred has joined #openstack-infra | 14:42 | |
| *** ryanpetrello has quit IRC | 14:42 | |
| *** achanda has quit IRC | 14:42 | |
| *** Shrews has joined #openstack-infra | 14:43 | |
| *** ryanpetrello_ has joined #openstack-infra | 14:43 | |
| *** ddieterl_ has quit IRC | 14:43 | |
| *** bhunter71 has joined #openstack-infra | 14:45 | |
| *** Longgeek has joined #openstack-infra | 14:45 | |
| *** poconosguy has joined #openstack-infra | 14:46 | |
| *** poconosguy has quit IRC | 14:46 | |
| *** dguerri is now known as _dguerri | 14:48 | |
| *** _dguerri is now known as dguerri | 14:48 | |
| *** shardy_ has joined #openstack-infra | 14:48 | |
| *** dguerri is now known as _dguerri | 14:49 | |
| *** _dguerri is now known as dguerri | 14:50 | |
| *** zhiwei has quit IRC | 14:50 | |
| *** zz_jgrimm is now known as jgrimm | 14:50 | |
| *** shardy has quit IRC | 14:50 | |
| mordred | sdague: so - create_userrc doesn't seem to honor all the same things as the things that wind up being set if you source openrc | 14:51 |
| mordred | sdague: for instance, identity api | 14:51 |
| mordred | sdague: for instance, identity api version | 14:51 |
| sdague | mordred: that's entirely possible, I'm actually complete oblivious to the create_userrc path | 14:51 |
| mordred | awesome | 14:51 |
| sdague | I've only ever used the openrc one | 14:51 |
| mordred | so openrc does not interact with the data written by create_userrc then? | 14:51 |
| *** freerunner has joined #openstack-infra | 14:52 | |
| sdague | dtroyer: ^^^ ?? | 14:52 |
| *** ddieterly has joined #openstack-infra | 14:52 | |
| dtroyer | nope. I don't recall who contributed create_userrc but we've never used it directly | 14:52 |
| *** fhubik_afk is now known as fhubik | 14:53 | |
| mordred | aweoms | 14:54 |
| *** shardy_ has quit IRC | 14:54 | |
| mordred | so I should fairly much ignore it | 14:54 |
| sdague | yeh, I think that's best | 14:54 |
| *** shardy has joined #openstack-infra | 14:54 | |
| jeblair | dtroyer, sdague: I refreshed https://review.openstack.org/167447 to remove the trove bits from devstack | 14:55 |
| *** cody-somerville has joined #openstack-infra | 14:55 | |
| *** cody-somerville has quit IRC | 14:55 | |
| *** cody-somerville has joined #openstack-infra | 14:55 | |
| *** armax has joined #openstack-infra | 14:55 | |
| *** yamahata has joined #openstack-infra | 14:55 | |
| sdague | mordred: how are you expecting to handle password in the general case | 14:55 |
| sdague | jeblair: cool, is the project-config change landed? | 14:55 |
| mordred | sdague: I'm not sure I understand your question? | 14:55 |
| sdague | I kind of lost track with the depends on spider | 14:55 |
| sdague | mordred: so.... whatever this is, needs the password for the account | 14:56 |
| mordred | yah | 14:56 |
| *** fawadkhaliq has joined #openstack-infra | 14:56 | |
| sdague | which seems to conflict with a previous statement of "have horizon export you a working one" | 14:56 |
| mordred | oh - so - sorry - two different things there | 14:56 |
| sdague | ok | 14:57 |
| *** ryanpetrello_ has quit IRC | 14:57 | |
| jeblair | sdague: the only non-merged changes are: 167447 - remove trove from devstack <- 168145 - Enable trove plugin for trove tempest job | 14:57 |
| jeblair | gah one more | 14:57 |
| *** ryanpetrello_ has joined #openstack-infra | 14:57 | |
| mordred | sdague: I imagine that for the import-thing-exported-by-horizon that we could have it prompt for a password if there is not one in the config - or maybe if there is a "prompt_for_password" flag in the exported config | 14:58 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for heat tox 2.0 bug 1455065 https://review.openstack.org/183074 | 14:58 |
| jeblair | <- 168146 Revert "Temporarily add trove plugin configuration" | 14:58 |
| openstack | bug 1455065 in heat "New tox version broke integration tests" [Critical,In progress] https://launchpad.net/bugs/1455065 - Assigned to Sirushti Murugesan (sirushtim) | 14:58 |
| mordred | (becaue it's possible the cloud may not use password auth, in which case prompting on missing password would be a mistake) | 14:58 |
| *** ryanpetrello_ is now known as ryanpetrello | 14:58 | |
| sdague | mordred: ok, cool | 14:58 |
| *** fhubik has quit IRC | 14:58 | |
| mordred | sdague: for devstack, I intend to write the password to the file :) | 14:59 |
| sdague | mordred: yep, for devstack that's fine | 14:59 |
| sdague | I was mixing with the other case | 14:59 |
| sdague | jeblair: ok, cool, I'm +2 on the remove, will let dtroyer +A | 14:59 |
| jeblair | sdague: ^ so 3 patches; we're using the new system already because of the temporary patch to devstack-gate (so it's ignoring what's set in project-config); once the devstack cleanup lands, we can merge the project-config change to set that explicitly, then remove the temporary patch to devstack-gate | 14:59 |
| sdague | jeblair: great | 14:59 |
| sdague | thanks | 14:59 |
| electrical | pelix: fungi is there a specific reason that reporters are only available in maven2-moduleset type jobs? and not freestyle for example | 15:01 |
| electrical | ? | 15:01 |
| *** dangers_away is now known as dangers | 15:01 | |
| *** achanda has joined #openstack-infra | 15:01 | |
| *** ryanpetrello has quit IRC | 15:01 | |
| *** ivar-lazzaro has joined #openstack-infra | 15:02 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for neutron/manila tox 2.0 bug 1455102 https://review.openstack.org/183075 | 15:02 |
| openstack | bug 1455102 in python-novaclient "some test jobs broken by tox 2.0 not passing env variables" [High,In progress] https://launchpad.net/bugs/1455102 - Assigned to Sean Dague (sdague) | 15:02 |
| *** ryanpetrello has joined #openstack-infra | 15:02 | |
| *** sks has quit IRC | 15:04 | |
| *** sergsh has quit IRC | 15:04 | |
| *** Longgeek has quit IRC | 15:04 | |
| sdague | fungi: so we had another burst on - http://status.openstack.org//elastic-recheck/#1282876 | 15:04 |
| *** Longgeek_ has joined #openstack-infra | 15:05 | |
| *** Longgeek_ has quit IRC | 15:05 | |
| *** achanda has quit IRC | 15:05 | |
| sdague | were there any other proposed fixes to deal with git hanging up on us? | 15:05 |
| *** Longgeek has joined #openstack-infra | 15:05 | |
| *** tiswanso has quit IRC | 15:06 | |
| *** dboik_ has quit IRC | 15:06 | |
| *** baoli has quit IRC | 15:06 | |
| *** ivar-lazzaro has quit IRC | 15:06 | |
| jeblair | sdague, fungi: http://logs.openstack.org/63/183063/1/check/gate-python-openstacksdk-pep8/8a03454/console.html | 15:07 |
| fungi | sdague: i'm not sure that's an accurate representation | 15:07 |
| jeblair | pulled a random from there ^ | 15:07 |
| fungi | sdague: looks like failures for other reasons are clouding that count | 15:07 |
| sdague | fungi: ok | 15:07 |
| fungi | sdague: that error appears if we retry | 15:07 |
| jeblair | dns resolution failure, followed by git failure.... looks like maybe abject network failure | 15:07 |
| fungi | sdague: then if the job fails for another reason, e-r thinks that was the cause | 15:07 |
| sdague | fungi: ok, so we need to update our signature | 15:08 |
| jeblair | and yeah, git eventually succeeded | 15:08 |
| *** tiswanso_ has joined #openstack-infra | 15:08 | |
| *** dboik has joined #openstack-infra | 15:08 | |
| fungi | in that example, "E303 too many blank lines (3)" | 15:08 |
| *** dustins_ has joined #openstack-infra | 15:08 | |
| mordred | devstack's support for keystone v3 seems ... not really a thing | 15:09 |
| mordred | like, I see no mention of USER_DOMAIN or PROJECT_DOMAIN anywhere | 15:10 |
| mordred | and v2.0 is hardcoded in many places | 15:10 |
| mordred | so, I'm guessing I should not worry too much about that for now? | 15:11 |
| sdake | mordred would you mind taking a look at https://review.openstack.org/#/c/182886/ | 15:11 |
| *** tnovacik has joined #openstack-infra | 15:11 | |
| sdague | yeh, interestingly, if we weren't retrying, this would have failed | 15:11 |
| *** akshai has quit IRC | 15:11 | |
| sdague | so... that's good | 15:11 |
| sdake | and jaeger too if hes awake ;) | 15:12 |
| mordred | sdake: it is marked WIP | 15:12 |
| mriedem | i just hit https://bugs.launchpad.net/openstack-ci/+bug/1383403 in the gate | 15:12 |
| openstack | Launchpad bug 1383403 in OpenStack Core Infrastructure "fatal: unable to access 'https://git.openstack.org/openstack-infra/devstack-gate/': gnutls_handshake() failed: A TLS packet with unexpected length was received." [Medium,Fix committed] | 15:12 |
| *** dustins has quit IRC | 15:12 | |
| mriedem | http://logs.openstack.org/55/182755/3/gate/gate-grenade-dsvm/f2ebe25/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-05-14_14_07_15_814 | 15:12 |
| *** dguerri is now known as _dguerri | 15:12 | |
| *** _dguerri is now known as dguerri | 15:12 | |
| sdake | groan | 15:12 |
| sdake | sec mordred | 15:12 |
| fungi | electrical: not really sure why they're maven only to be honest. hopefully pelix or zaro know | 15:12 |
| *** dboik has quit IRC | 15:13 | |
| *** tiswanso_ has quit IRC | 15:13 | |
| *** craigbr has joined #openstack-infra | 15:13 | |
| sdague | mriedem: now that retry logic is in there - http://status.openstack.org//elastic-recheck/#1282876 seems no longer specific enough, we might need to prune that back | 15:14 |
| *** arxcruz has quit IRC | 15:14 | |
| mriedem | sdague: yes, i'm working on that | 15:14 |
| *** bswartz has quit IRC | 15:14 | |
| sdague | cool | 15:14 |
| fungi | sdague: mriedem: though also looks like devstack-gate doesn't retry `git remote prune origin` calls | 15:14 |
| mriedem | sdague: alas, i have to run to get a haircut b/c i need to look beautiful for the summit | 15:14 |
| dtroyer | mordred: devstack and v3 are incubating…the first flag to turn off v2 is about to merge, morgan is working on the rest of a v2-free setup | 15:14 |
| mordred | ok | 15:15 |
| openstackgerrit | Debo~ Dutta proposed openstack-infra/project-config: Cognitive - Machine Learning as a Service https://review.openstack.org/182886 | 15:15 |
| mordred | dtroyer: also, there seems to be no support for glance v2, is that right? | 15:15 |
| mordred | I mean, Im asking because there seems to be support for nova and cinder api version switching, even though nova and cinder do not have more than one version, and there is no support for switching versions for keystone and glance which do ... so it's a bit wat? | 15:15 |
| dtroyer | in devstack? I don't recall anything explicit either way, just glance's defaults | 15:16 |
| mriedem | fungi: i can add a retry loop for git remote prune origin | 15:16 |
| sdake | ok mordred debo fixed that up | 15:16 |
| *** bswartz has joined #openstack-infra | 15:16 | |
| mordred | dtroyer: awesome | 15:17 |
| sdague | mordred: so... basically that's a reflection of the involvment rate of the various teams | 15:17 |
| sdague | cinder does have more than one version | 15:18 |
| nfedotov | Hello all! For some reason it is impossible to clone repos from git.openstack.org via git:// protocol " Connection reset by peer". Does somebody experiences similar issue? | 15:18 |
| mordred | sdague: really? | 15:18 |
| mordred | sdague: well, good on them for doing it right, beause so far it's never caused me pain :) | 15:19 |
| sdague | yeh, v1 isn't yet removed. openstackclient doesn't actually work with cinder v2 api yet | 15:19 |
| BobBall | What's the easiest way to stop executing jobs in the zuul/jenkins setup while continuing to queue jobs to run? | 15:19 |
| mordred | Shrews, greghaynes: also - see scrollback with me and sdage a little bit ago about passwords and importing files from horizon | 15:20 |
| fungi | BobBall: you can put your jenkins into prepare-for-shutdown | 15:20 |
| BobBall | OK - wasn't sure if that would break other things :) Thanks, | 15:20 |
| greghaynes | mordred: yea, there was convo late last night about tox2 being to blame | 15:20 |
| fungi | BobBall: all running jobs will complete, no new jobs will run, zuul will queue them up waiting for jenkins to come back online | 15:20 |
| sdague | fungi: hmmm... there is some spiking - http://logs.openstack.org/07/183007/7/check/gate-tempest-dsvm-neutron-src-python-novaclient/2a12c04/console.html | 15:21 |
| BobBall | Perfect. | 15:21 |
| sdague | and http://logs.openstack.org/07/183007/7/check/check-novaclient-dsvm-functional/740c9da/console.html | 15:21 |
| sdague | the git server at high load atm? | 15:21 |
| *** skraynev has quit IRC | 15:21 | |
| sdague | that just faceplanted on 2 jobs at about the same time on a clone | 15:22 |
| mordred | greghaynes: yup. we have a short-term workaroudn for tox2 - and are looking at just making things work from clouds.yaml longer term | 15:22 |
| BobBall | And that's why I just stopped the libvirt+xen ci sdague :) | 15:22 |
| fungi | BobBall: zuul could probably be easily extended to do something similar. right now it has a signal handler callout to wait for changes being tested to report but queue any new trigger events in preparation to save its queues and exit. could probably add a pair of signals to quiesce and resume using most of the existing logic | 15:22 |
| sdague | BobBall: ok, it's you? :) | 15:23 |
| mordred | sdague, greghaynes, Shrews, dtroyer: https://review.openstack.org/183080 | 15:23 |
| *** skraynev has joined #openstack-infra | 15:23 | |
| BobBall | I'm happy for you to say it's my fault... but I'm afraid I have no idea what's going on :) | 15:23 |
| BobBall | Just that it seems to be failing a huge percentage of the changes starting about 30 minutes ago ish for me | 15:24 |
| *** links has quit IRC | 15:24 | |
| BobBall | And the best part is that because 'logs' doesn't exist it fails to upload _anything_ for me! :) | 15:25 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch on docs and publish jobs for magnum https://review.openstack.org/182998 | 15:25 |
| *** ihrachyshka_ has quit IRC | 15:26 | |
| sdague | mrodden: I think you have to fix at least one thing | 15:27 |
| sdague | sorry mordred ^^ | 15:27 |
| mrodden | sdague, what did i break? | 15:27 |
| mrodden | :) | 15:27 |
| sdague | there is at least 1 typo, 1 ... I'm not sure that's safe, and then 1 of me giving you hard time on naming | 15:27 |
| *** sabeen1 has joined #openstack-infra | 15:28 | |
| mordred | sdague: awesome | 15:28 |
| greghaynes | mordred: also commented | 15:28 |
| mordred | sdague: the auth dict is because of keystone auth plugins | 15:29 |
| mordred | sdague: so taht you can do validation of parameters | 15:29 |
| mordred | sdague, greghaynes: awesome. thanks! updated | 15:31 |
| mordred | sdague: and yes - it's a bit bonghits | 15:31 |
| *** nfedotov has quit IRC | 15:32 | |
| *** sabeen3 has joined #openstack-infra | 15:32 | |
| *** linuxmaniac has quit IRC | 15:34 | |
| *** sabeen1 has quit IRC | 15:34 | |
| Shrews | mordred: commented. i think you mean ! -f for the clouds.yaml | 15:34 |
| nibalizer | fungi: cool, did everything with afs go okay? | 15:35 |
| jeblair | BobBall, fungi, sdague: starting around 14:20, the git fe's have seen an unusually high number of established tcp connections http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=2417&rra_id=0&view_type=tree&graph_start=1431605641&graph_end=1431617713&graph_height=120&graph_width=500&title_font_size=10 | 15:36 |
| fungi | nibalizer: seems to not have failed to puppet anyway | 15:36 |
| mordred | Shrews: doh | 15:36 |
| Shrews | mordred: i'd even go with ! -e | 15:36 |
| fungi | jeblair: nodepool image updates? | 15:37 |
| fungi | jeblair: they kick off at 14:14 | 15:37 |
| jeblair | fungi: ah maybe so then | 15:37 |
| mordred | Shrews: ! -e ? why? | 15:37 |
| *** jcoufal has quit IRC | 15:37 | |
| jeblair | dib will make this better :) | 15:37 |
| fungi | jeblair: might also not be our nodepool image updates, if other people reuse our nodepool.yaml example verbatim | 15:37 |
| mordred | jeblair: yah it will | 15:37 |
| Shrews | mordred: -e tests if it exists, -f tests if it's a file, not directory | 15:37 |
| mordred | Shrews: fair | 15:38 |
| Shrews | mordred: just more logical | 15:38 |
| BobBall | fungi: The number of connections is way higher than earlier in the month though | 15:38 |
| fungi | BobBall: yeah, that's why i saw it may not be ours | 15:38 |
| fungi | s/saw/say/ | 15:38 |
| clarkb | fungi: BobBall could also be more reliable clouds/image builds | 15:38 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Testing the ceph job. https://review.openstack.org/183092 | 15:38 |
| *** linuxmaniac has joined #openstack-infra | 15:38 | |
| clarkb | for example bare-centos6 was not building for ~6 weeks | 15:38 |
| * BobBall changes his nodepool.yaml to kick off the image builds at a different time to spread the load :) | 15:39 | |
| *** nadya has joined #openstack-infra | 15:40 | |
| *** nadya is now known as Guest79156 | 15:41 | |
| *** Guest79156 is now known as _nadya_ | 15:41 | |
| *** Swami has joined #openstack-infra | 15:42 | |
| fungi | mordred: 20008 connections through git-fe01 from hound.openstack.org so far this hour... normal? | 15:42 |
| sdague | mordred: also... for this to work in the novaclient test case, we're going to need admin creds in there as well | 15:42 |
| fungi | er, sorry, that was last hour (14:00-15:00 utc) | 15:42 |
| mtreinish | fungi: oh, that would probably explain why I can't clone trying to spin up a new devstack node | 15:43 |
| fungi | and another 5120 from it through fe02 | 15:45 |
| fungi | so it made 25128 connections in an hour (a little over 7 per second) | 15:46 |
| jeblair | root@hound:/var/log# less hound.log | 15:47 |
| jeblair | "hound.log" may be a binary file. See it anyway? | 15:47 |
| *** _nadya_ has quit IRC | 15:47 | |
| sdake | redirect loop at libertydesignsummit.sched.org | 15:48 |
| * sdake groans | 15:48 | |
| *** sarob has joined #openstack-infra | 15:48 | |
| jeblair | sdake: wfm? | 15:48 |
| sdake | restarting chrome | 15:49 |
| sdake | ya it keeps going to linked in and back to libertydesignsummit | 15:49 |
| sdake | i got an error about linked in not authorized earlier as well | 15:49 |
| sdake | maybe thats the cause | 15:49 |
| fungi | weird. not for me in firefox anyway | 15:49 |
| jeblair | sdake: oh, yeah, i'm not linking anything in. | 15:50 |
| sdake | I mean I have my linkedin credentails set I think | 15:50 |
| *** markvoelker has quit IRC | 15:50 | |
| sdake | but maybe there is a temporary breakage | 15:50 |
| nibalizer | also "ERROR:open /home/hound/data/idx-403a35fe9c764811/raw/LICENSE: too many open files" from the hound ui | 15:50 |
| *** gyee has joined #openstack-infra | 15:51 | |
| fungi | i didn't realize there was a linkedin integration in sched.org, but it wouldn't surprise me. i don't have a linkedin account/profile so probably would never have noticed it | 15:51 |
| jeblair | so if this were a real server, we'd have diagnostic information in cacti, etc. :/ | 15:51 |
| fungi | jeblair: i'm in favor of just turning it off for now and mordred can investigate when he's free | 15:52 |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras and env markers https://review.openstack.org/181795 | 15:52 |
| fungi | since it's a non-production demo/test anyway | 15:52 |
| jeblair | lemme lsof first... | 15:52 |
| electrical | fungi: okay. looking forward to know :-) | 15:53 |
| wznoinsk | hi, git clone http://git.openstack.org/openstack/requirements.git works, while git clone git://git.openstack.org/openstack/requirements.git doesn't, using same socks proxy, could someone check their end? | 15:54 |
| *** dtantsur is now known as dtantsur|afk | 15:54 | |
| *** yamamoto has quit IRC | 15:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Mark apps-catalog py34 job as non-voting https://review.openstack.org/183019 | 15:55 |
| fungi | jeblair: mordred: the count of connections works out to about 38 per repo we host per hour, so unsurprising if it's trying to remote update copies of all of them periodically (not sure how many separate git protocol connections that entails) | 15:55 |
| *** ajmiller has joined #openstack-infra | 15:55 | |
| *** yamamoto has joined #openstack-infra | 15:55 | |
| *** ildikov has quit IRC | 15:55 | |
| sdake | groan redirct loop on both sched sites for summit | 15:56 |
| fungi | it's possible these are very short-lived and not contributing heavily to traffic volume or backend load | 15:56 |
| *** dims has quit IRC | 15:56 | |
| *** clif_h has quit IRC | 15:56 | |
| sdake | sent tweet to support hopefully they cna fix | 15:56 |
| jeblair | mordred, fungi, nibalizer: houndd had 1312 files like /home/hound/data/idx-2dbe7b1ea98063a5/tri open. i stopped it. | 15:56 |
| fungi | jeblair: thanks. let's see if that gets any better | 15:56 |
| *** dims has joined #openstack-infra | 15:56 | |
| fungi | wznoinsk: are you sure you have access to reach port 9418/tcp on the internet? | 15:57 |
| wznoinsk | fungi, it wasn't working for last 1h and started to work a second before your msg ;-) | 15:58 |
| mordred | jeblair, fungi: ah, fun - so yeah, it sets up a poller thread to check for changes in the repos - we can lower the frequency in config | 15:58 |
| fungi | wznoinsk: okay, then that's the overload we're invesitgating | 15:58 |
| jeblair | fungi: i think it just git pulls frequently, so it should be fairly lightweight. it also did not log any errors around the times we're interested, though it did see some earlier. | 15:58 |
| fungi | wznoinsk: in the future "stopped working" conveys a lot more information than "doesn't work" (which could also be interpreted as "never worked") | 15:59 |
| jeblair | mordred: i think it's not a problem under normal circumstances; i think at the moment we're wondering if it's related to the spike. it may not be. | 15:59 |
| *** markvoelker has joined #openstack-infra | 15:59 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Switch to environments in tripleo-heat-templates. https://review.openstack.org/183098 | 15:59 |
| wznoinsk | true, will keep that in mind | 15:59 |
| *** kmartin has joined #openstack-infra | 15:59 | |
| jeblair | mordred: however, nibalizer did point out an error potentially related to leaking fds | 15:59 |
| mordred | jeblair: k. it also could be made more efficient - it currently doesn't grok watching local repos for updates - which I think it shoudl learn. if it did, then we could easily have an rsync cron job instead of a per-repo poller | 16:00 |
| fungi | wznoinsk: in that case, yes we're tracking a spike in activity for git protocol access to our git.openstack.org farm | 16:00 |
| mordred | nibalizer: ah - lovely | 16:00 |
| mordred | nibalizer: wanna learn go? | 16:00 |
| *** puranamr has joined #openstack-infra | 16:00 | |
| mordred | sdague: re: admin creds - overriding username/password is easy to do and I'll include that in my novaclient patch | 16:01 |
| jeblair | fungi: i see an increase in established tcp corresponding with a decrease in tcp opens on fe02 | 16:02 |
| *** craigbr has quit IRC | 16:02 | |
| *** jlanoux_ has quit IRC | 16:02 | |
| mordred | sdague: in the gate case, admin/admin and demo/demo are the admin and non-admin users, yeah? | 16:02 |
| *** ivar-lazzaro has joined #openstack-infra | 16:02 | |
| *** yfried has joined #openstack-infra | 16:03 | |
| nibalizer | mordred: uh | 16:03 |
| *** tonytan4ever has joined #openstack-infra | 16:03 | |
| nibalizer | mordred: are we all learning go? | 16:03 |
| mordred | nibalizer: NO | 16:03 |
| Clint | rust, all rust | 16:04 |
| mordred | Clint: ++ | 16:04 |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Update networking-ovn job https://review.openstack.org/183100 | 16:05 |
| sigmavirus24 | rust++ | 16:05 |
| *** asrangne has joined #openstack-infra | 16:05 | |
| clarkb | heh if you keep throwing rust and go around I will have to dust off my haskell | 16:05 |
| jeblair | snap | 16:06 |
| mordred | clarkb: just learn rust - it's got the good bits of haskell without the pedantic | 16:06 |
| clarkb | mordred: those are the good bits of haskell (and no rust really doesn't have the good bits of haskell) | 16:06 |
| greghaynes | hah | 16:06 |
| mordred | clarkb: we may differ in what we think are the good bits then :) | 16:06 |
| fungi | jeblair: so the next highest offenders are https not git protocol. an ip address in hpecore.net (6029) and a bunch of addresses spread through the top half of a /24 in ibm.com (5955) for that hour | 16:07 |
| *** ivar-lazzaro has quit IRC | 16:07 | |
| *** freerunner has quit IRC | 16:07 | |
| jeblair | given a log entry in haproxy, can i tell what source port it used when connecting to the backend? | 16:08 |
| clarkb | fungi: jeblair fwiw I do think that it is the expensive to pack operation that hurts more than total connections, but git's logging isn't really good enough to track that down and prove it | 16:08 |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Update networking-ovn job https://review.openstack.org/183100 | 16:08 |
| clarkb | jeblair: iirc yes but I need to go look at an haproxy log really quickly | 16:08 |
| mordred | yah - I mean, this is one of the reasons that kernel.org started offering public rsync iirc | 16:08 |
| *** aswadr has quit IRC | 16:08 | |
| mordred | because they hit scaling problems doing git things | 16:08 |
| *** tiswanso has joined #openstack-infra | 16:09 | |
| *** dboik has joined #openstack-infra | 16:09 | |
| pleia2 | good morning | 16:10 |
| nibalizer | pleia2: o/ | 16:10 |
| clarkb | jeblair: wow you know what, it looks like no | 16:10 |
| clarkb | how can a proxy server not do that | 16:11 |
| clarkb | (I am reading the haproxy loggging doc section now to confirm) | 16:11 |
| jeblair | yeah, i can't see a way :( | 16:11 |
| *** puranamr has quit IRC | 16:11 | |
| *** puranamr has joined #openstack-infra | 16:12 | |
| sigmavirus24 | clarkb: fwiw, I'm down with haskell too | 16:13 |
| sigmavirus24 | especially given haskell and rust at least have half-way decent package managers | 16:13 |
| *** tiswanso has quit IRC | 16:13 | |
| *** dboik has quit IRC | 16:14 | |
| clarkb | it has been too long since I wrote any haskell, I should find and excuse to brush up | 16:14 |
| clarkb | but breakfast now | 16:14 |
| *** tqtran has joined #openstack-infra | 16:14 | |
| fungi | clarkb: i thought xmonad kept you on your haskell toes? | 16:14 |
| clarkb | fungi I occasionally have to fix my config. trying it on tumbleweed was fun | 16:15 |
| *** shardy has quit IRC | 16:16 | |
| greghaynes | clarkb: you took the tumble? | 16:17 |
| *** freerunner has joined #openstack-infra | 16:17 | |
| *** unicell1 has quit IRC | 16:17 | |
| clarkb | on a vm | 16:17 |
| clarkb | had to make xmonad work | 16:18 |
| *** e0ne is now known as e0ne_ | 16:18 | |
| zaro | morning | 16:19 |
| jeblair | fungi: gitfe02 has something like 28 connections from rax HK | 16:20 |
| fungi | oh, looking at concurrent connections as opposed to number initiated over a span of time? | 16:22 |
| jeblair | fungi: yeah. they seem to be very long-lived | 16:22 |
| *** patrickeast has joined #openstack-infra | 16:22 | |
| zaro | electrical: that might be a jenkins limitation. | 16:23 |
| pelix | electrical: I believe those just happened to be the early precursors to publishers and were only compatible with maven project types and were called reporters since that's how maven would refer to them. | 16:23 |
| pelix | Subsequently as the idea of publishers was created to be generic and plugins moved to use that naming as they were made compatible with other project types | 16:23 |
| *** craigbr has joined #openstack-infra | 16:23 | |
| *** derekh_ has quit IRC | 16:24 | |
| *** afazekas has quit IRC | 16:24 | |
| *** dkranz has quit IRC | 16:25 | |
| *** e0ne_ is now known as e0ne | 16:25 | |
| fungi | jeblair: oh, i see. over ipv6 for git protocol | 16:27 |
| jeblair | ya | 16:27 |
| btully | are there any current/known issues with the openstack git repos? gettling lots of timeouts and very slow connections | 16:27 |
| fungi | btully: yes, something is hammering our git server farm hard and we're trying to track it down | 16:29 |
| clarkb | connection reset by peer is the error ya? I have checked the bw use on the frontends and that is below our theoretical limits | 16:29 |
| clarkb | but rax sending resets for us was the old behavior that seems to be related to bw limiting | 16:29 |
| jogo | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZmF0YWw6IHJlYWQgZXJyb3I6IENvbm5lY3Rpb24gcmVzZXQgYnkgcGVlclwiIEFORCB0YWdzOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzMTYyMDc2MTM4MywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ== | 16:29 |
| jogo | its been happening for about 2 hours | 16:29 |
| fungi | jogo: yep, that matches our graphs | 16:29 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for heat tox 2.0 bug 1455065 https://review.openstack.org/183074 | 16:29 |
| openstack | bug 1455065 in heat "New tox version broke integration tests" [Critical,In progress] https://launchpad.net/bugs/1455065 - Assigned to Sirushti Murugesan (sirushtim) | 16:29 |
| clarkb | in that case I had confirmed that the connection that was reset on the jenkins slave side never showed up in the haproxy logs | 16:29 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for neutron/manila tox 2.0 bug 1455102 https://review.openstack.org/183075 | 16:29 |
| openstack | bug 1455102 in python-novaclient "some test jobs broken by tox 2.0 not passing env variables" [High,In progress] https://launchpad.net/bugs/1455102 - Assigned to Sean Dague (sdague) | 16:30 |
| *** gokrokve_ has joined #openstack-infra | 16:30 | |
| *** whoops has joined #openstack-infra | 16:30 | |
| *** cody-somerville has quit IRC | 16:30 | |
| electrical | zaro: pelix okay afaik all reporters work with freestyle ad well. I can run some tests if you guys want. | 16:30 |
| *** dannywil_ has joined #openstack-infra | 16:30 | |
| *** krtaylor has quit IRC | 16:31 | |
| jeblair | clarkb, fungi, jogo: the logstash lines are _all_ from hpcloud nodes, none from rax | 16:31 |
| jeblair | perhaps the fault is not at the git farm | 16:31 |
| btully | thanks fungi - good luck tracking the offenders down! | 16:31 |
| jogo | jeblair: there is *one* from rax bare-trusty-rax-dfw-2552642 | 16:31 |
| jogo | 1 out of 24 | 16:32 |
| jogo | 25 | 16:32 |
| fungi | yes, entirely possible it's some connectivity problem, but wondering why people are seeing issues via git:// and not http(s):// | 16:32 |
| *** ashleighfarnham has joined #openstack-infra | 16:32 | |
| SpamapS | greghaynes: are you working on a devstack patch to emit clouds.yaml? | 16:33 |
| *** gokrokve has quit IRC | 16:33 | |
| jeblair | oh, if you go back further, there are more hits from rax | 16:33 |
| *** gokrokve_ has quit IRC | 16:34 | |
| greghaynes | SpamapS: Theres a hacky patch already, im working on osc support | 16:34 |
| *** yamamoto has quit IRC | 16:34 | |
| fungi | memory usage on git05 is pretty bad, and git01 is also bad but not as bad | 16:35 |
| SpamapS | greghaynes: ok, because it sounds like that would be a better answer to our functional test things than trying to ram through the OS_x vars we want. Yes? | 16:35 |
| greghaynes | SpamapS: https://review.openstack.org/183080 | 16:35 |
| greghaynes | SpamapS: yes | 16:35 |
| greghaynes | SpamapS: thats the plan | 16:35 |
| clarkb | fungi: it has 1GB free | 16:35 |
| clarkb | thouh that seems to rapidly change | 16:36 |
| greghaynes | SpamapS: basically, do ^ now, give osc a command to edit clouds.yaml based on env vars / args | 16:36 |
| clarkb | fungi: several git processes all using >500MB of ram | 16:36 |
| fungi | clarkb: right, but from a performance standpoint that means basically no caching in memory | 16:36 |
| clarkb | lsof reports its for nova | 16:37 |
| SpamapS | greghaynes: I have no idea what that means. | 16:37 |
| fungi | not surprising | 16:37 |
| *** dkranz has joined #openstack-infra | 16:37 | |
| mordred | SpamapS: osc == python-openstackclient | 16:37 |
| mordred | SpamapS: we're going to add cloud config manipulation commands to it | 16:37 |
| fungi | i'll see if filtering logs by hits to git05 paints any different of a picture | 16:37 |
| SpamapS | mordred: how does that help us get shade's functional testing working? | 16:38 |
| mordred | SpamapS: that know how to transform either env vars, or files of env vars into the clouds.yaml | 16:38 |
| zaro | electrical, pelix : maybe try to approve both https://review.openstack.org/#/c/182988/ & https://review.openstack.org/#/c/165628/ and then do a release today? | 16:38 |
| mordred | SpamapS: well, we rework shade's functional testing to consume clouds.yaml instead of env vars | 16:38 |
| SpamapS | since we've decided not to rollback tox 2.0 incompatible changes (very frustrating IMO) | 16:38 |
| mordred | SpamapS: first using the devstack patch above - but then we clean up the devstack support by using the osc command | 16:38 |
| *** craigbr has quit IRC | 16:38 | |
| *** claudiub has joined #openstack-infra | 16:39 | |
| yolanda | jeblair , mordred, can you give some feedback about https://review.openstack.org/170788 ? the nodepool queues change | 16:39 |
| *** wenlock has joined #openstack-infra | 16:39 | |
| mordred | SpamapS: because we can provide this functionality to other people and not just our gate | 16:39 |
| yolanda | today we had some hpcloud slowness and it should have been helpful | 16:39 |
| *** BharatK has quit IRC | 16:39 | |
| fungi | jeblair: that rax hk address seems to be spreading its concurrent sessions across both git-fe lbs | 16:39 |
| mordred | SpamapS: not sure I follow the rollback frustration? I'm probably also frustrated though | 16:39 |
| *** jlanoux has joined #openstack-infra | 16:39 | |
| zaro | electrical: if tox isn't working for you it might be that you need to install a compatible version of tox and virtualenv | 16:40 |
| SpamapS | mordred: the patch to cap tox at <2.0 has three -1's, mostly with the theory that "it's early in the cycle so its ok for everything to be broken" | 16:40 |
| mordred | SpamapS: well, pinning tox is a big deal, beause it means "pip install tox; tox -epy27" will be broken for people | 16:40 |
| fungi | SpamapS: experience has shown that if we hardcode a specific piece of tooling which is outside our self-testing loop, it's a massive undertaking to undo that later | 16:40 |
| mordred | so it's just masking a real world breakage | 16:40 |
| SpamapS | which frustrates me because our cycle has very little to do with our downstreams' cycles and breaking all tests right now means we are discouraging them from tracking trunk. :-/ | 16:40 |
| SpamapS | fungi: yes, we are faced with two evils. :) | 16:41 |
| SpamapS | I choose the massive undertaking later, because later means we can plan, track, and execute it on a set schedule. | 16:41 |
| mordred | SpamapS: they'd have to also track our infra puppet to pin tox for us pinning tox to help them | 16:41 |
| SpamapS | Now means "zomg its on fire haaallp" | 16:41 |
| SpamapS | mordred: devstack is responsible for installing tox :-/ | 16:41 |
| fungi | SpamapS: the options are 1. hide the broken from everyone and hope they fix it anyway with no real incentive to do so, or 2. leave the broken obvious so they can tell when they've fixed it all | 16:42 |
| mordred | SpamapS: only for devstack | 16:42 |
| *** jistr has quit IRC | 16:42 | |
| *** unicell has joined #openstack-infra | 16:42 | |
| SpamapS | fungi: "no real incentive" isn't true. We can set a date, say, L2, where it will be unpinned. Give people time to plan, work on fixes, and maybe even have an experimental job which unpins it. | 16:42 |
| jeblair | fungi, clarkb: i looked for session termination anomolies, and they only ones i see around the times we're looking at are a handful of "cL", which is "client timeout while sending the last data after the server has finished" | 16:42 |
| *** asilenkov_ is now known as asilenkov | 16:42 | |
| SpamapS | mordred: luckily, devstack is the primary victim of the env var change. | 16:43 |
| fungi | SpamapS: as i said, for much of our infrastructure that's outside our self-testing loop. we build new worker images and hope everyone's covered their bases | 16:43 |
| mordred | SpamapS: well, I believe I've got patches up to fix or provide vehicle for fixing all the things - although I need to do a shade one real quick | 16:44 |
| SpamapS | OK, I guess I don't really grok the relationship we have with the things we test. I'll pipe down and just go back to zomg its on fire and haaaalp myself. ;) | 16:44 |
| *** enikanorov2 has quit IRC | 16:44 | |
| fungi | SpamapS: for a lot of our tests, low-level plumbing like setuptools, pip, virtualenv, tox, et cetera aren't easily testable in automation. they're preinstalled for use _by_ tests | 16:45 |
| fungi | they could be made testable, but that too is a major undertaking and i'd be hesitant to just freeze our entire toolchain on the assumption that someone will come along and solve that eventually | 16:46 |
| clarkb | jeblair: show errors against the haproxy admin socket doesn't show anything for either haproxy server | 16:46 |
| clarkb | jeblair: and it is surprisingly difficuly to map a connection on a backend back up to the frontend to see what is causing it (trying to see if I can figure out if these high memory nova packs are consistently from one location) | 16:46 |
| fungi | SpamapS: which means the devstack fix boils down to "use an older tox in devstack than we do everywhere else" and that comes with its own problems | 16:47 |
| SpamapS | fungi: so, I see what you're saying. We don't have an easy way to have a "future" toolchain. | 16:47 |
| fungi | SpamapS: not now at least. would be nice! | 16:47 |
| mordred | yah. I have thoughts ... | 16:47 |
| *** ildikov has joined #openstack-infra | 16:47 | |
| clarkb | fungi: the other issue we run into is when people run tox for unittest they get latest tox and it doesn't work | 16:47 |
| SpamapS | Seems like something that will come up so infrequently as to not be a priority. | 16:47 |
| clarkb | fungi: this resulted in tons of questions every day when we had pinned tox | 16:47 |
| clarkb | "why does tox not work" | 16:48 |
| SpamapS | Yeah, it's two evils, I completely acknowledge that. | 16:48 |
| mordred | SpamapS: btw: https://review.openstack.org/183106 is what I just put together for novaclient functional testing - I think it probably wants to be made into a fixture - but for now cargo culting is probably fine | 16:48 |
| pleia2 | it's really not obvious to folks which one to use when it's pinned | 16:48 |
| zaro | clarkb: check this out, https://groups.google.com/d/msg/repo-discuss/gnxUdzyzbIY/RX28u6V4P3kJ | 16:48 |
| SpamapS | And without an actual way to provide both evils to users, picking "most current software evil" is probably the right play. | 16:48 |
| fungi | yep. for the devstack case maybe that's less of a problem, but it's not really teh "devstack" case it's the "tempest" case so you're effectively saying "let's fix tempest by pinning tox in devstack, nobody uses tempest except by running it within a devstack-built environment" | 16:49 |
| clarkb | zaro: neat, so they are using jenkins now instead of that builder script? | 16:49 |
| zaro | clarkb: i'm wondering if it might be better to use that gerritforge setup to get our gerrit plugins instead of building it ourselves. | 16:49 |
| clarkb | zaro: or maybe using both I guess | 16:49 |
| SpamapS | pleia2: well .. we could put these things in a file in global reqs and advise people to install only those versions. | 16:49 |
| clarkb | zaro: thats a possibility | 16:49 |
| SpamapS | But again, as fungi points out, you need a way to _ALSO_ have automated testing of unpinned, or people will never unpin. | 16:49 |
| mordred | clarkb, zaro: why would we use someone else's build farm? | 16:50 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/project-config: Make neutron-dvr job voting at the gate https://review.openstack.org/180230 | 16:50 |
| mordred | oh neat though! they use jjb | 16:50 |
| clarkb | mordred: becusae we don't need to build all the gerrit plugins (or really even gerrit itself) | 16:50 |
| mordred | clarkb: fair enough | 16:50 |
| clarkb | mordred: thats mostly an artifact of history | 16:50 |
| jeblair | we do have a local patch | 16:50 |
| zaro | i think we still need to build gerrit since we do a fork. | 16:50 |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| clarkb | jeblair: so doing more lsof digging, nova does seem to consistently need that memory overhead | 16:51 |
| mordred | does anyone else think that it's funny that gerrit.gerritforge.com is a jenkins and not a gerrit? | 16:51 |
| zaro | but the plugins should be ok to just grab from there instead of building it ourselves | 16:51 |
| SpamapS | mordred: yes that should go into oslo.test me thinks. | 16:51 |
| SpamapS | or oslo_test I suppose is what we call things now | 16:51 |
| clarkb | SpamapS: even with automated testing of unpinned I doubt many would care to fix | 16:51 |
| jeblair | clarkb: yes, i have observed this in the past; a given be can only handle a small number of nova clones at a time. do you think the be's are overtaxed at the moment? | 16:51 |
| clarkb | jeblair: ya I think that may be the case, then as they fall behind haproxy may be telling clients togo away | 16:52 |
| zaro | haha! i didn't even think about that but, yeah what's up with that? would be a good question for luca | 16:52 |
| *** signed8bit is now known as signed8bit_ZZZzz | 16:52 | |
| jeblair | clarkb: i don't see haproxy doing that though | 16:52 |
| clarkb | jeblair: we do have connection limits particularly on the git:// protocol because the git daemon itself does not do any connection management | 16:52 |
| clarkb | jeblair: ah in that case likely not the problem | 16:52 |
| fungi | jeblair: clarkb: maybe not a great correlation, but those established connections from rax hk died down and so did memory consumption on the backends to which they were being shunted | 16:52 |
| clarkb | jeblair: though I want ot say the limits are above what we see on the backends | 16:53 |
| clarkb | iirc we would peak at ~30 connections per backend during normalish operations | 16:53 |
| clarkb | *concurrent on average as reported by cacti | 16:53 |
| jeblair | clarkb: i would expect that to show up as " s. " for the connection state (that's a regex match anything dot) | 16:53 |
| jeblair | clarkb: and that only shows up 6 times today, all well before the current issue | 16:54 |
| jeblair | (sD to be precise in these cases) | 16:54 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1697&rra_id=all that graph has ramped up over time, but its still quite low overall | 16:54 |
| *** fawadk has joined #openstack-infra | 16:55 | |
| fungi | jeblair: clarkb: also comparing the netstat output to the haproxy logs, that wasn't one ip address from rax hk, it was more like 60 | 16:55 |
| fungi | netstat was truncating a bunch of addresses from the same /64 | 16:56 |
| jeblair | fungi: oh i was using lsof and i saw 28 simultaneous when i took a snapshot | 16:56 |
| SpamapS | clarkb: I disagree that nobody would care to fix. One simply needs to give them a clear time frame when the pin will be advanced, and get it on their roadmap. | 16:56 |
| electrical | zaro: your patch looks good. | 16:56 |
| clarkb | jeblair: reading docs I agree, s. and likely sD or sQ | 16:56 |
| SpamapS | clarkb: just like python 3, now that EOL for 2 is imminent, people are actually doing it. | 16:57 |
| clarkb | SpamapS: our classic example is the last time we pinned tox fwiw | 16:57 |
| fungi | jeblair: from the same singleton address? got it | 16:57 |
| clarkb | SpamapS: I had to fix it, and it was like pulling teeth | 16:57 |
| SpamapS | clarkb: Perhaps you should not have fixed it? | 16:57 |
| clarkb | SpamapS: I had to for python34 support | 16:57 |
| electrical | zaro: that should make everything work nicely on Python 3.x | 16:57 |
| *** yamahata has quit IRC | 16:58 | |
| *** ddieterly has quit IRC | 16:58 | |
| jeblair | clarkb, fungi: we've seen those addresses for the past few days at least, so it's not like they just started up today | 16:58 |
| SpamapS | Open a bug in everything that's broken, target it at CURRENT_MILESTONE+1, and note in the bug that the pin will be advanced immediately following CURRENT_MILESTONE+1 | 16:58 |
| fungi | yep, may also be unrelated to this | 16:58 |
| *** gokrokve has joined #openstack-infra | 16:58 | |
| *** fawadkhaliq has quit IRC | 16:59 | |
| *** ddieterly has joined #openstack-infra | 16:59 | |
| *** enikanorov2 has joined #openstack-infra | 17:00 | |
| clarkb | SpamapS: the amount of managerial effort involved in that is o_O and for a large majority of the projects you will end up wedging things just as badly as you do if you let the broken in immediately or worse ecause instead of one broken you have accumulated 10 brokens that all need fixing | 17:00 |
| clarkb | SpamapS: the accumulated broken was part of the tox issue last time | 17:00 |
| electrical | yolanda: is there anything i can do / test to make the reporters open for other type of jobs? | 17:00 |
| jeblair | fungi, clarkb: also, the HK addresses transferred more data yesterday and the day before, and had more connections. | 17:01 |
| jeblair | they transferred 37610899052 today (so far) and 60251402389 yesterday | 17:01 |
| * jeblair just learned: |awk '{s+=$11} END {print s}' | 17:01 | |
| fungi | ooh! a list sum function? | 17:02 |
| pleia2 | hah, nice | 17:02 |
| jeblair | yeah, sum column 11 in haproxy log | 17:02 |
| fungi | i've always resorted to hacking separate perl or other madness | 17:02 |
| fungi | that's handy | 17:02 |
| pelix | zaro: found a chunk of bugs in python-jenkins around python 3 and post requests. Have some additions to the tests that force checking on the request to ensure it's valid on python 3. | 17:03 |
| pelix | Most of the api calls just needed to have their calls changed from using '' to trigger a POST opperation, to use b'' instead | 17:03 |
| *** mestery has joined #openstack-infra | 17:03 | |
| electrical | pelix: interesting. i didn't get that far so was unable to test :-) | 17:04 |
| pelix | but for create_job and reconfig_job the tests currently don't encode what is being sent in. Should we assume the application will correctly encode the XML, or always test and encode it if not already encoded? | 17:04 |
| *** krtaylor has joined #openstack-infra | 17:04 | |
| *** yamamoto has quit IRC | 17:05 | |
| electrical | we could always call encode just to be safe? | 17:05 |
| yolanda | electrical, there was a spec for rewriting a bit of jjb | 17:05 |
| SpamapS | clarkb: I agree with all that you say, however, an additional factor is that the fixes for the 10 brokens can be done on a plan, where as right now the brokens must be done _immediately_ and interrupt any plans. | 17:05 |
| *** e0ne has quit IRC | 17:05 | |
| yolanda | also you can send patches to add new modules for jjb | 17:05 |
| clarkb | jeblair: more info on the large nova pack processes, they seem to be IO bound with a ton of writes to fd 1, I think thats basically writing up a pipe back to the parent process which is speaking tcp | 17:05 |
| *** sdake has quit IRC | 17:06 | |
| pelix | electrical: yeah, just decided to see if some of the other api calls worked on python 3 when testing your patch and discovered that Request have different enough behaviour that it was worth modifying the tests to build the request fully to check for exceptions. | 17:06 |
| electrical | yolanda: i did send that logstash reporter module for jjb. but reporters are limited to maven2 type jobs but i use freestyle mainly | 17:06 |
| clarkb | jeblair: but 8192 bytes at a time very slowly | 17:06 |
| electrical | pelix: ahh i see :-) | 17:06 |
| *** sdake has joined #openstack-infra | 17:06 | |
| *** freerunner has quit IRC | 17:07 | |
| *** wenlock has quit IRC | 17:07 | |
| jeblair | clarkb: are you sure it's not the reads that are slow? | 17:07 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1715&rra_id=all | 17:07 |
| clarkb | jeblair: this process isn't doing any reads (it may be the other end reading slowly off the pipe) | 17:07 |
| *** pabelanger has quit IRC | 17:08 | |
| jeblair | clarkb: something is reading off of disk, right? | 17:08 |
| clarkb | jeblair: but I think that may be why it uses so much memory, the entire thing is loaded into memory, then its writing it out | 17:08 |
| jeblair | clarkb: okay i see | 17:08 |
| jeblair | clarkb: so you think it's already read substantial data, and is now slowly streaming it out | 17:08 |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Modify tests to exercise Request on python3 https://review.openstack.org/183111 | 17:08 |
| jeblair | clarkb: (possibly limited by client bandwidth); and in the interim, it's sitting with the entire nova pack in memory? | 17:08 |
| clarkb | jeblair: yup, and while it does that slowly it has consumed that memory resource | 17:08 |
| clarkb | jeblair: ya | 17:09 |
| pelix | zaro: see change ^ for test changes, need to fix up a few things first | 17:09 |
| zaro | electrical: if you haven't notice, i -1 your python-jenkins change :) | 17:09 |
| electrical | zaro: i saw yeah. want me to add the decode there as well ? | 17:09 |
| fungi | electrical: pelix: "always call encode just to be safe" introduces new problems, in particular trying to run a unicode-encoded string through .encode('utf-8') raises a UnicodeDecodeError exception | 17:09 |
| *** pabelanger has joined #openstack-infra | 17:09 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Do not automatically target bugs for stable releases https://review.openstack.org/179447 | 17:09 |
| electrical | fungi: uhg true. | 17:09 |
| zaro | electrical: i don't have python 3 setup. so yeah, just test to see if it's needed. | 17:10 |
| *** ianw has quit IRC | 17:10 | |
| fungi | electrical: er, a utf-8 encoded string through .encode('utf-8') | 17:10 |
| electrical | zaro: okay, will try if i can find it out and report back | 17:10 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Decouple openstack_project::server from Gerrit https://review.openstack.org/182840 | 17:10 |
| jeblair | clarkb: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1698&rra_id=all | 17:10 |
| *** dboik has joined #openstack-infra | 17:10 | |
| jeblair | clarkb: it looks like it got close to using all the ram, but not quite? | 17:10 |
| jeblair | clarkb: the machine has not swapped | 17:11 |
| clarkb | jeblair: I did see an madvise go by too, so it may be freeing up part of that buffer as its write pointer moves along. It did a MADV_DONTNEED | 17:11 |
| *** shashankhegde has quit IRC | 17:11 | |
| fungi | electrical: basically to successfully use .encode() you need to first know or be able to somehow detect the existing encoding, if any | 17:11 |
| clarkb | jeblair: ya, I don't think it every completel ran out of memory but the pressure did affect the memory cache size | 17:12 |
| fungi | so that you can decode back to unicode and translate into the new encoding | 17:12 |
| electrical | fungi: makes sense. found this: http://stackoverflow.com/a/4987414/975001 | 17:12 |
| clarkb | jeblair: we may be trading pack files cached by kernel for pack file in $process memory space, which negatively affects the next pack process? | 17:13 |
| *** gokrokve has quit IRC | 17:13 | |
| AJaeger | I just noticed on the nova queue that job check-tempest-dsvm-multinode-full is NOT_REGISTERED. The job runs on node devstack-trusty-2-node. Anybody an idea what's wrong here? | 17:13 |
| *** gokrokve has joined #openstack-infra | 17:13 | |
| fungi | electrical: yep, that's a fairly reasonable way to do it, or use the various string handling utility functions from six | 17:14 |
| electrical | fungi: ah. okay. not so familiar with python yet ;-) | 17:14 |
| electrical | fungi: zaro i have to run and go out for dinner. will do some more checking / testing tomorrow. feel free to ping me on IRC, ill be able to read back | 17:14 |
| electrical | or email :-) | 17:14 |
| *** dboik has quit IRC | 17:14 | |
| fungi | electrical: https://pythonhosted.org/six/ | 17:15 |
| *** dboik has joined #openstack-infra | 17:15 | |
| *** dboik has quit IRC | 17:15 | |
| electrical | fungi: ah interesting :-) | 17:15 |
| fungi | convenience library to handle writing code compatible with both python 2 and 3 | 17:15 |
| *** dboik has joined #openstack-infra | 17:16 | |
| jeblair | clarkb, fungi: it makes sense that lots of slow clients cloning nova would run up the ram and established connections; i don't actually see how that results in connection errors for other clients :/ | 17:16 |
| *** gyee has quit IRC | 17:16 | |
| *** freerunner has joined #openstack-infra | 17:16 | |
| *** baoli has joined #openstack-infra | 17:16 | |
| *** cdent has quit IRC | 17:17 | |
| clarkb | jeblair: maybe apache failed to fork/startthread and its nicely closing the connection for new clients? except I think most of the issues were over git protocol ya? | 17:17 |
| *** gokrokve has quit IRC | 17:17 | |
| zaro | fungi: so i cannot repro https://code.google.com/p/gerrit/issues/detail?id=3365 on my local machine. | 17:17 |
| fungi | clarkb: yeah, it was apparently git protocol which saw the brunt of the impact | 17:17 |
| *** gokrokve has joined #openstack-infra | 17:18 | |
| clarkb | I am not seeing anything inthe apache error log that would support that theory | 17:18 |
| zaro | fungi: it is happending on review-dev. now i'm wondering if it's something to do with the reverse proxy setup. i don't have that setup on my local machine. | 17:18 |
| *** tiswanso has joined #openstack-infra | 17:19 | |
| fungi | zaro: it's possible. i tried several ways of working around it in the proxy config, to no avail. but it definitely seems to be a behavior difference between 2.8 and 2.10 | 17:19 |
| jeblair | clarkb, fungi: i do note some packet loss from my workstation to fe02 | 17:19 |
| *** Krinkle|detached is now known as Krinkle | 17:20 | |
| jeblair | and 01 | 17:20 |
| zaro | fungi: behavior difference + some config on our gerrit servers | 17:21 |
| *** yamahata has joined #openstack-infra | 17:21 | |
| jeblair | clarkb, fungi: i'm browsing the logstash hits and they seem to be mostly on initial connections for 'git remote update' | 17:21 |
| fungi | jeblair: clarkb: looking at the logs on git05, git-daemon definitely shows a log of "Disconnected (with error)" along with things like "git-upload-pack died of signal 13" | 17:22 |
| *** bhuvan has joined #openstack-infra | 17:22 | |
| fungi | er, a lot of | 17:22 |
| zaro | fungi: would it be possible to turn off the reverse proxy thing on review-dev for a few minutes to see if it still a problem? | 17:22 |
| clarkb | fungi: syslog? | 17:22 |
| fungi | clarkb: messages | 17:22 |
| clarkb | oh right rhel | 17:22 |
| fungi | (this is sparta^H^H^H^H^H^Hrhel!) | 17:22 |
| btran_ | Hi my Cinder CI testbed was running fine about a week ago. But now I"m getting an problem with devstack gate getting configured and setup. | 17:23 |
| jeblair | clarkb, fungi: i have to run for a bit; the graphs and logstash seem to indicate that whatever was happening is over | 17:23 |
| btran_ | here are my logs, the error occurs at the end of the logs: http://ftp.nimblestorage.com/openstack_ci/iscsi/178992/6/check/nimble-iscsi-driver-dsvm-volume/1665/logs/devstack-gate-setup-workspace-new.txt | 17:23 |
| *** abregman_ has quit IRC | 17:23 | |
| asselin | btran_, are you reusing nodes? | 17:23 |
| fungi | zaro: the proxy is what's making gerrit reachable. you'd need to reconfigure gerrit to listen on a public address on 443 and add https to it somehow, or maybe just have it listen on port 80 | 17:23 |
| btran_ | asselin: sometimes yes, is that my problem. I need to run in clean setup all the time? | 17:24 |
| asselin | btran_, yes | 17:24 |
| asselin | btran_, seems the leftovers of a previous test is causing the script to fail | 17:24 |
| clarkb | in other news the ES cluster is green and happy again | 17:25 |
| *** dizquierdo has quit IRC | 17:25 | |
| *** gokrokve has quit IRC | 17:26 | |
| btran_ | asselin: i see.... is anyone working on fixing that? or will this script cleanup problem be here for some time? | 17:26 |
| *** gokrokve has joined #openstack-infra | 17:26 | |
| asselin | btran_, everyone is using clean nodes. take a look at nodepool: http://docs.openstack.org/infra/system-config/nodepool.html | 17:26 |
| asselin | btran_, so yes, the script cleanup problem will be there for a long time.... | 17:28 |
| fungi | btran_: the problem will always exist. if you're running jobs with root privileges you're running untrusted code that could make the entire system inoperable/unusable. someone could just propose a patch that deletes everything on the root filesystem | 17:28 |
| zaro | fungi: ok. i'll give it a try. | 17:28 |
| btran_ | asselin: yes that's our goal to setup nodepool, but currently there are some resource issues, thanks for the info | 17:28 |
| btran_ | fungi: ok thanks for data/info, I will rise the issue with my mgmt | 17:29 |
| clarkb | arg puppet lint hates me | 17:30 |
| fungi | btran_: you don't have to use nodepool necessarily, but it's the way we and a lot of other ci systems have solved the "use a fresh worker for every job" problem | 17:30 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Pass OS_ variables through to functional tests https://review.openstack.org/183121 | 17:30 |
| SpamapS | mordred: ^ if we land this we can do the clouds.yaml fixes decoupled from "all the other things" | 17:31 |
| SpamapS | greghaynes: ^ | 17:31 |
| SpamapS | Shrews: ^ | 17:31 |
| SpamapS | clarkb: ^ | 17:31 |
| *** gokrokve has quit IRC | 17:31 | |
| greghaynes | SpamapS: Does that work in pre 2.0? | 17:31 |
| greghaynes | as in, not explode | 17:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-logstash: Logrotate the indexer.log logfile. https://review.openstack.org/182894 | 17:31 |
| *** achanda has joined #openstack-infra | 17:32 | |
| greghaynes | hrm, looks like thats documented in 1.9 | 17:32 |
| AJaeger | fungi, clarkb, jeblair: I noticed on the nova queue that job check-tempest-dsvm-multinode-full is NOT_REGISTERED. The job runs on node devstack-trusty-2-node. Doy you have an idea what's wrong? The job has been around for a longer time, just renamed a month ago. | 17:32 |
| greghaynes | oh, says new in 2.0 | 17:32 |
| fungi | AJaeger: checking now | 17:32 |
| SpamapS | greghaynes: yes | 17:32 |
| SpamapS | passenv has been there for a while | 17:33 |
| SpamapS | just not turned on by default | 17:33 |
| btran_ | fungi: yea, I could probably just deploy a fresh worker vm or revert to a clean snapshot of the vm for now, until we deploy/implement nodepool. thanks | 17:33 |
| greghaynes | ah | 17:33 |
| SpamapS | hm | 17:33 |
| SpamapS | I just saw that you can do OS_* | 17:33 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for d-g git prune bug 1455123 https://review.openstack.org/183122 | 17:33 |
| SpamapS | seems like cheating.. | 17:33 |
| openstack | bug 1455123 in OpenStack-Gate "fatal: unable to access 'https://git.openstack.org/openstack/sahara/': GnuTLS recv error (-9): A TLS packet with unexpected length was received." [Undecided,In progress] https://launchpad.net/bugs/1455123 - Assigned to Matt Riedemann (mriedem) | 17:33 |
| SpamapS | but might be pragmatic given the short life of the patch | 17:33 |
| *** Apsu has quit IRC | 17:33 | |
| dims | AJaeger: adding check-requirements does not seem to trigger gate-magnum-requirements job (http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n1838) i am looking at zuul status for a WIP patch 183112 | 17:34 |
| *** Apsu has joined #openstack-infra | 17:34 | |
| AJaeger | dims: let me check... | 17:34 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Pass OS_ variables through to functional tests https://review.openstack.org/183121 | 17:35 |
| AJaeger | dims, check http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n883 | 17:36 |
| AJaeger | let me adjust your WIP patch to trigger it... | 17:36 |
| *** shashankhegde has joined #openstack-infra | 17:37 | |
| SpamapS | I'm sad pypy isn't compelling enough for us to keep on top of it. :-( | 17:37 |
| clarkb | SpamapS: I thought I fixed it (anyother thing I fixed because no one else was willing to ...) | 17:38 |
| dims | thanks AJaeger | 17:38 |
| SpamapS | clarkb: see sdague's message. Nobody is tending to it. | 17:38 |
| clarkb | SpamapS: have examples of failures? I can track those down and see if I am missing image builds or something like that | 17:38 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate: Add a loop to git_prune https://review.openstack.org/183124 | 17:38 |
| clarkb | hrm I seriously just fixed it on monday | 17:38 |
| SpamapS | clarkb: you shouldn't need to fix it. THats the point. | 17:38 |
| AJaeger | dims: check your change again ;) | 17:38 |
| EmilienM | jeblair: after reading the doc and code, I still have a question: is 'Depends-on' able to test a patchset from another project? | 17:38 |
| fungi | SpamapS: i can see the attraction, but then again i can also understand wanting to focus on having an openstack that works well on one python interpreter before trying to make it work well on more than one | 17:38 |
| SpamapS | clarkb: no fixers is a sign that it isn't in anybody's critical path. | 17:38 |
| dims | AJaeger: aha! :) | 17:39 |
| AJaeger | dims: now zuul shows me "gate-magnum-requirements:" | 17:39 |
| SpamapS | fungi: err... two interpreters... (2.7 and 3.4+ ;-) | 17:39 |
| clarkb | SpamapS: ya | 17:39 |
| AJaeger | dims, so everything is fine... | 17:39 |
| dims | whew thanks for the tip | 17:39 |
| SpamapS | also as the swift guys pointed out.. it helps, but not enough to scale to where they want to scale. | 17:39 |
| AJaeger | dims: you're welcome! | 17:39 |
| *** kmartin has quit IRC | 17:39 | |
| fungi | SpamapS: in some ways, still three for a bit (2.6 for some stable branches), and if you consider how every distro modifies the python they provide it's actually a different python interpreter per platform we test on | 17:40 |
| clarkb | SpamapS: well you have to turn off JIT with eventlet in pypy so its pretty unfun for our needs | 17:40 |
| openstackgerrit | Sirushti Murugesan proposed openstack-infra/devstack-gate: Add Heat to features.yaml https://review.openstack.org/183126 | 17:40 |
| clarkb | SpamapS: if however we stopped eventleting that could potentially get much much better | 17:40 |
| fungi | then again if we stopped eventleting even normal cpython has the potential for significant speedups | 17:41 |
| SpamapS | clarkb: the few things left that really need eventlet (read: non wsgi) are the things that would get the most benefit from pypy. :-/ | 17:41 |
| clarkb | (or fixed eventlet in pypy I suppose, but unsure if this is a fundamental issue with greentheads running outside of the interpreters thread model or something that is fixable) | 17:41 |
| SpamapS | It's probably a fundamental incompatibility between pypy's magic and eventlet's magic. | 17:41 |
| *** harlowja has quit IRC | 17:41 | |
| *** harlowja has joined #openstack-infra | 17:42 | |
| EmilienM | jeblair: nevermind, I misread it and it works | 17:42 |
| clarkb | fungi: any luck with the 2-node trusty gearman registration? | 17:42 |
| SpamapS | Anyway, I think there's an axiom forming in my head about this. "You only get one magical thing in your software." or something like that. | 17:43 |
| erlon | krtaylor: ping | 17:43 |
| clarkb | EmilienM: if you use depends-on with a change in another project zuul will include that change in your stack of zuul refs for your job | 17:43 |
| EmilienM | clarkb: ack, thanks | 17:43 |
| clarkb | EmilienM: this is why figuring out how to set up deps properly with beaker is important for us | 17:44 |
| *** bhunter71 has quit IRC | 17:44 | |
| clarkb | EmilienM: I think we can do it with zuul-cloner being called by that beaker setup script thing | 17:44 |
| fungi | AJaeger: clarkb: looking to see what node type zuul thinks this job should require, but it's definitely not registered since the restart | 17:44 |
| clarkb | fungi: should be devstack-trusty-2-node | 17:45 |
| *** bhunter71 has joined #openstack-infra | 17:45 | |
| mordred | SpamapS: yah - totally. I think the "pass through env vars for shade functional" is definitely step 1 | 17:45 |
| fungi | clarkb: we have one ready since before the zuul restart. i'll delete it | 17:45 |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:45 | |
| fungi | in theory the next one that builds will get the job registered in zuul | 17:46 |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:46 | |
| *** kmartin has joined #openstack-infra | 17:46 | |
| clarkb | fungi sounds good, thanks | 17:46 |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:46 | |
| AJaeger | thanks, fungi ! | 17:46 |
| fungi | i'll let you know as soon as it's showing registered again | 17:47 |
| fungi | there's one building currently | 17:47 |
| AJaeger | dims, what are you doing with 183112 now? I'll ignore any review requests now ;) | 17:48 |
| clarkb | fungi: we might also consider slightly bumping the ready node count for that flavor since nova is using it in check now | 17:48 |
| fungi | clarkb: that's a good idea, yes | 17:48 |
| *** johnthetubaguy_ has joined #openstack-infra | 17:49 | |
| EmilienM | clarkb: I'm asking because in our Puppet jobs, it's pulling from master even if you have Depends-on thing | 17:49 |
| *** bhunter71 has quit IRC | 17:49 | |
| EmilienM | clarkb: I can show you an example | 17:50 |
| EmilienM | clarkb: https://review.openstack.org/#/c/181299/ | 17:50 |
| crinkle | actually our tests specifically do a git clone from master, we don't have a way to co-gate them | 17:50 |
| fungi | crinkle: you could replace git clone with zuul-cloner | 17:50 |
| dims | AJaeger: haha, i am all set don't worry about that one now. the requirements check job probably version checks only when things change i guess. testing that theory | 17:50 |
| *** johnthetubaguy has quit IRC | 17:50 | |
| *** johnthetubaguy_ is now known as johnthetubaguy | 17:50 | |
| crinkle | fungi: how does that work, and would that work for a user running tests locally? | 17:51 |
| *** bhunter71 has joined #openstack-infra | 17:51 | |
| clarkb | EmilienM: crinkle I know that was one of my dislikes of how beaker is currently set up | 17:51 |
| clarkb | I need to distill the current feedback on that spec and write one with work items to address the beaker things | 17:51 |
| fungi | crinkle: it 1. knows how to inspect the calling environment for ref names passed in via job parameters but also 2. takes advantage of the local git cache for other projects on our workers | 17:51 |
| fungi | crinkle: clarkb has a patch proposed to make zuul-cloner work even without zuul envvars present and fall back to normal git behaviors | 17:52 |
| crinkle | cool | 17:52 |
| fungi | but alternatively you could just add some conditional logic around the presence/absence of a ZUUL_REF in the env | 17:52 |
| clarkb | fungi: crinkle EmilienM ya thats the change that should let us use it in beaker as it should work on your laptop and in the gate | 17:53 |
| fungi | and use zuul-cloner when it's present but regular git clone somebranch when it's not | 17:53 |
| EmilienM | clarkb: nice | 17:53 |
| *** e0ne has joined #openstack-infra | 17:53 | |
| *** marcusvrn has quit IRC | 17:54 | |
| fungi | clarkb: AJaeger: check-tempest-dsvm-multinode-full is registered in zuul now | 17:54 |
| *** teran_ has quit IRC | 17:55 | |
| *** marcusvrn has joined #openstack-infra | 17:55 | |
| *** teran has joined #openstack-infra | 17:55 | |
| AJaeger | great, thanks fungi! | 17:57 |
| *** HeOS has quit IRC | 17:58 | |
| adam_g | fungi, you mentioned earlier this week that the $gerrit_patch->$lp_project(s) postback is configurable somewhere. any pointers on where that lives? trying to replicate neutron's flow where patches to multiple neutron-* projects update bugs on the single neutron project | 17:59 |
| AJaeger | fungi: and already in use ;) | 17:59 |
| *** teran has quit IRC | 17:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Capitalize references to resoruces https://review.openstack.org/171370 | 18:00 |
| fungi | adam_g: yes, you can set a groups: neutron parameter on each of them in openstack-infra/project-config:gerrit/projects.yaml | 18:00 |
| dims | AJaeger: thanks for setting up the transifex and helping with bootstrapping the projects for translations! | 18:01 |
| adam_g | fungi, ah! thanks | 18:01 |
| *** BharatK has joined #openstack-infra | 18:02 | |
| *** dannywilson has joined #openstack-infra | 18:02 | |
| krtaylor | erlon, pong | 18:02 |
| erlon | krtaylor: Hi Kurt | 18:02 |
| *** jlanoux has quit IRC | 18:03 | |
| erlon | krtaylor: I was looking for that cross section on summit, about CI integration but couldn't fund | 18:03 |
| *** jlanoux has joined #openstack-infra | 18:03 | |
| erlon | *find | 18:03 |
| *** patrickeast has quit IRC | 18:03 | |
| *** patrickeast_ has joined #openstack-infra | 18:03 | |
| erlon | krtaylor: did you manage to get the session? | 18:03 |
| *** ivar-lazzaro has joined #openstack-infra | 18:04 | |
| *** tonytan4ever has quit IRC | 18:04 | |
| *** dannywil_ has quit IRC | 18:05 | |
| krtaylor | erlon, no, but we do have an 2 infra working sessions and a BoF | 18:07 |
| krtaylor | erlon, see the links in the meeting agenda | 18:08 |
| erlon | krtaylor: ill check | 18:08 |
| AJaeger | dims: since you went through the exercise, what about documenting some of it at https://wiki.openstack.org/wiki/Translations ? Could you do this, please? | 18:09 |
| dims | AJaeger: that's the least i can do :) am on it | 18:10 |
| mordred | clarkb: zuul-cloner as locally usable was the next hard thing you were going to tackly, right? | 18:10 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add db api method to get uuid from test_id https://review.openstack.org/180741 | 18:11 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add db api method to get per status counts for a test https://review.openstack.org/180742 | 18:11 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Group akanda sub-projects https://review.openstack.org/183146 | 18:12 |
| erlon | krtaylor: whats is the title/day? | 18:12 |
| *** hdd has joined #openstack-infra | 18:12 | |
| *** SotK has quit IRC | 18:12 | |
| clarkb | mordred you mean for installation? if so yes | 18:13 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Decouple openstack_project::server from Gerrit https://review.openstack.org/182840 | 18:14 |
| *** freerunner has quit IRC | 18:14 | |
| mordred | clarkb: neat | 18:15 |
| *** dimtruck is now known as zz_dimtruck | 18:15 | |
| krtaylor | erlon, BoF ->https://libertydesignsummit.sched.org/event/454bd293910d345dc9fe6f3b438f2bbc | 18:15 |
| clarkb | mordred I was just thinking about what a pip install zuul[cloner] might do to indtall only that tools deps and install only that entrypoint | 18:15 |
| clarkb | but that may be a terrible idea | 18:15 |
| mordred | clarkb: no - I think tat's a GREAT idea | 18:16 |
| *** zz_dimtruck is now known as dimtruck | 18:16 | |
| clarkb | kk | 18:16 |
| krtaylor | erlon, Infra work session -> https://libertydesignsummit.sched.org/event/2fc3cbea0ca84febf5a334cd5ca4d723 | 18:16 |
| erlon | krtaylor: thanks | 18:16 |
| *** sabeen3 has quit IRC | 18:16 | |
| clarkb | that means entrypoints need rxtra like defs too | 18:16 |
| clarkb | and extras need to be subtractive too | 18:17 |
| clarkb | so not super simple | 18:17 |
| *** sabeen1 has joined #openstack-infra | 18:17 | |
| AJaeger | dims: I guess you better enhance https://wiki.openstack.org/wiki/Translations/Infrastructure | 18:17 |
| krtaylor | erlon, sure, np | 18:17 |
| *** spzala has joined #openstack-infra | 18:18 | |
| *** gyee has joined #openstack-infra | 18:19 | |
| *** freerunner has joined #openstack-infra | 18:23 | |
| clarkb | can I get eyes on https://review.openstack.org/182894 and https://review.openstack.org/182895 those are bug fixes for log leaking in the logstash/ES system that cropped up yesterday with full /s | 18:23 |
| openstackgerrit | Merged openstack-infra/shade: Pass OS_ variables through to functional tests https://review.openstack.org/183121 | 18:24 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Allow JJB to set connection timeout to jenkins server https://review.openstack.org/151472 | 18:24 |
| *** ildikov has quit IRC | 18:25 | |
| pleia2 | clarkb: to be clear, the ES stuff rotates itself, but logcheck does not? | 18:26 |
| mordred | clarkb: tchaypo is working on adding extras to reqs - excellent point about entrypoints though - that's an interesting problem | 18:27 |
| clarkb | pleia2 ya | 18:28 |
| pleia2 | s/logcheck/logstash | 18:28 |
| pleia2 | too many logs | 18:28 |
| pleia2 | thanks | 18:28 |
| flaper87 | Do we use cloud archive's PPA in our gates? IIRC, the gate policy is to not use PPAs, am I right ? | 18:30 |
| *** claudiub has quit IRC | 18:30 | |
| mordred | flaper87: that is right | 18:31 |
| mordred | flaper87: we do not use its PPA | 18:31 |
| clarkb | we did try at one time but it didnt work | 18:31 |
| flaper87 | mordred: clarkb thank you! | 18:32 |
| *** tonytan4ever has joined #openstack-infra | 18:32 | |
| AJaeger | flaper87: could you approve some translation imports, please: https://review.openstack.org/179630 and https://review.openstack.org/176633? | 18:32 |
| flaper87 | AJaeger: of course | 18:33 |
| AJaeger | dolphm: what about https://review.openstack.org/#/c/179331/ ? I just rechecked and would appreciate an import, please | 18:34 |
| AJaeger | thanks, flaper87 | 18:34 |
| flaper87 | AJaeger: done | 18:34 |
| flaper87 | np | 18:34 |
| *** jlvillal has quit IRC | 18:36 | |
| dims | AJaeger: i tried to doc a really simple step by step for an existing project (like mine was) here - https://wiki.openstack.org/wiki/Translations/quickstart | 18:37 |
| dims | if you like it we can move it to one of the other pages. just wanted to get the formatting right hence started a fresh page | 18:38 |
| *** ildikov has joined #openstack-infra | 18:38 | |
| AJaeger | dims: shall I move it and update on the fly? | 18:39 |
| *** otter768 has joined #openstack-infra | 18:40 | |
| AJaeger | dims: thanks! | 18:40 |
| *** baoli has quit IRC | 18:40 | |
| AJaeger | oh, there's even some initial content - let's merge with https://wiki.openstack.org/wiki/Translations#Integrating_translations_into_your_project | 18:41 |
| AJaeger | your section complements nicely what's there... | 18:41 |
| *** jlvillal has joined #openstack-infra | 18:41 | |
| AJaeger | and some bits there are outdated ;( | 18:42 |
| lifeless | sdague: sorry, it was late and all: http://logs.openstack.org/98/182898/1/check/check-shade-dsvm-functional-neutron/d16eecf/ | 18:42 |
| *** fawadk has quit IRC | 18:42 | |
| *** atan8 has joined #openstack-infra | 18:43 | |
| *** Sukhdev has joined #openstack-infra | 18:43 | |
| *** BharatK has quit IRC | 18:44 | |
| *** otter768 has quit IRC | 18:45 | |
| sdague | lifeless: yep, no prob. So typically "gate is wedged" means tempest or something equivalent is blocking all jobs (or vast majority) | 18:46 |
| *** changbl has joined #openstack-infra | 18:46 | |
| lifeless | sdague: k | 18:47 |
| sdague | this was a lot narrower, so being more specific would be good, especially as a lot of base patches were passing fine | 18:47 |
| lifeless | sdague: so if you look at the backscroll | 18:47 |
| lifeless | we got Kennan and SpamapS reporting it | 18:47 |
| *** _nadya_ has joined #openstack-infra | 18:48 | |
| lifeless | late my local time, witching period US | 18:48 |
| lifeless | I'll try to be clearer in future | 18:48 |
| lifeless | but I figured *something* out on the list would help diagnosis when more folk showed up | 18:48 |
| sdague | yep, so that's fine. I just didn't feel like from your email on the list I had enough to go with. So just trying to figure out what would be better next time. | 18:49 |
| lifeless | yeah | 18:49 |
| *** fawadkhaliq has joined #openstack-infra | 18:49 | |
| lifeless | agree on having a bug; I should have put the effort into file one | 18:49 |
| sdague | And I think links to a few different failing jobs, and what classes of jobs are failing, would be really helpful | 18:49 |
| sdague | yeh, even without the bug, examples are good | 18:49 |
| lifeless | I had a mental hat on of 'responding' rather than 'filing', which was a box I think | 18:49 |
| *** Sukhdev has quit IRC | 18:50 | |
| *** maishsk_afk has joined #openstack-infra | 18:51 | |
| AJaeger | dims, I'm on it... | 18:51 |
| sdague | because I had a disconnect with "there's a thing, and this is patch" and that patch randomly failed grenade, which made me think this was going to require chaining. So after about 40 minutes of starting to set up the chaining patches, I got confused with the gate basically being green | 18:51 |
| sdague | even though the patch didn't land | 18:52 |
| *** _nadya_ has quit IRC | 18:52 | |
| *** yamamoto has joined #openstack-infra | 18:53 | |
| openstackgerrit | Yathiraj Udupi proposed openstack-infra/project-config: Added create permission to release group for project nova-solver-scheduler https://review.openstack.org/183167 | 18:53 |
| *** ildikov has quit IRC | 18:54 | |
| *** markvoelker has quit IRC | 18:54 | |
| *** dannywilson has quit IRC | 18:54 | |
| openstackgerrit | Alexandre Viau proposed openstack-infra/project-config: New stackforge project: puppet-surveil https://review.openstack.org/183168 | 18:57 |
| AJaeger | dims, check https://wiki.openstack.org/wiki/Translations#Python_Projects_.28General.29 again | 18:57 |
| sdague | anyway, it's fine now, people are working through point patches to get individual projects up | 18:57 |
| openstackgerrit | Alexandre Viau proposed openstack-infra/project-config: Surveil ACL: pushSignedTag requires surveil-release https://review.openstack.org/183170 | 18:58 |
| dims | AJaeger: "extract_messages" section is not really needed, defaults should work fine | 18:58 |
| *** yamamoto has quit IRC | 18:59 | |
| dims | right? (not sure) | 18:59 |
| AJaeger | dims: not sure either ;( If it works without, remove the content ;) | 19:00 |
| *** markvoelker has joined #openstack-infra | 19:00 | |
| dims | AJaeger: let's let it be for now :) | 19:00 |
| openstackgerrit | Alexandre Viau proposed openstack-infra/project-config: surveil-specs: use surveil acl https://review.openstack.org/183172 | 19:00 |
| *** maishsk_afk is now known as maishsk | 19:01 | |
| dims | AJaeger: reads well. thanks! | 19:01 |
| *** harlowja has quit IRC | 19:03 | |
| AJaeger | dims, thanks to your quickstart! | 19:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions https://review.openstack.org/183176 | 19:04 |
| *** harlowja has joined #openstack-infra | 19:05 | |
| *** dannywilson has joined #openstack-infra | 19:05 | |
| *** claudiub has joined #openstack-infra | 19:05 | |
| sdake | lots of create: patches going into gerrit acls ;-) | 19:06 |
| sdake | I wonder if thats missing from the documentation | 19:06 |
| *** hashar has joined #openstack-infra | 19:06 | |
| fungi | sdake: we didn't originally document it because it's only needed if you're going to create branches instead of having the openstack release management team do it and up until a couple months ago none of the projects outside the integrated release and libraries wanted stable branches | 19:07 |
| AJaeger | sdake: please review https://review.openstack.org/181646 | 19:07 |
| sdake | ajaeger willd o | 19:08 |
| fungi | sdake: the sudden rash of all projects wanting to create additional branches was unexpected | 19:08 |
| *** Krinkle is now known as Krinkle|detached | 19:08 | |
| AJaeger | fungi, 181646 documents it... | 19:08 |
| fungi | yep | 19:08 |
| AJaeger | (me is just fishing for reviews ;) | 19:09 |
| *** ildikov has joined #openstack-infra | 19:09 | |
| sdake | fungi ya just adding comedy relief ;) | 19:09 |
| fungi | indeed! | 19:09 |
| sdake | ajaeger looks good | 19:11 |
| *** baoli has joined #openstack-infra | 19:11 | |
| clarkb | fungi I think its related to openstack libs doing it | 19:12 |
| AJaeger | thanks, sdake ! | 19:12 |
| *** shashankhegde has quit IRC | 19:12 | |
| clarkb | but stackforge isnt required to | 19:12 |
| sdake | ajaeger say one of our guys wants to get this in before summit so we can announce in our 2pm tuesday session on kolla that we actuallly do real CI :) any chancey ou can review? | 19:12 |
| sdake | https://review.openstack.org/#/c/182927/ | 19:12 |
| clarkb | also not really here niw waiting to see what verdict on summit travel is | 19:12 |
| *** sc68cal has quit IRC | 19:12 | |
| *** johnsom_ has joined #openstack-infra | 19:13 | |
| sdake | I think the syntax is right, but I'm not certain | 19:13 |
| *** baoli has quit IRC | 19:14 | |
| *** baoli has joined #openstack-infra | 19:14 | |
| *** BObH_ has joined #openstack-infra | 19:15 | |
| greghaynes | clarkb: do we need to bribe the dr so theyll say you should go? | 19:15 |
| clarkb | maybe but I doubt it will work | 19:15 |
| lifeless | clarkb: wat, you missing the summit? | 19:15 |
| fungi | sdake: so... do you only want it to fail if tox -e images exits nonzero? | 19:15 |
| sdake | pro tip - you can buy codeine OTC in canada ;) | 19:16 |
| *** smarcet has quit IRC | 19:16 | |
| BObH_ | Hey folks, looking for a pointer. | 19:16 |
| sdake | fungi right, if tox -e images exits with 0 that means we were successful I hope :) | 19:16 |
| clarkb | lifeless dunno yet, depends on whether or not twins are cooperating | 19:16 |
| lifeless | clarkb: twins - eeek | 19:17 |
| *** annegentle has quit IRC | 19:17 | |
| clarkb | lifeless indeed | 19:17 |
| fungi | sdake: but if say tox -e setupenv errors you still want to run tox -e images | 19:17 |
| sdake | I have two kids already | 19:17 |
| sdake | yikes ;) | 19:17 |
| sdake | if setupenv errors it should exit there | 19:17 |
| *** johnsom_ has left #openstack-infra | 19:17 | |
| sdake | it should exit on each line | 19:17 |
| sdake | i was thinking it needed && but wasn't clear | 19:17 |
| sdake | I am not a pro on infra unfortunately like you guys :( | 19:17 |
| fungi | sdake: okay, then you probably need set -e but also the quoting is wrong. i'll leave comments | 19:18 |
| BObH_ | Who do I contact about this when I try to get to gerrit? https://review.openstack.org/#SigninFailure, SIGN_IN< contackt+site+administrator when signing in to review.openstack.org? | 19:18 |
| sdake | fungi thanks alot ! :) | 19:18 |
| fungi | BObH_: i don't think i've ever seen that. what's your launchpad username? | 19:19 |
| BObH_ | hansenb | 19:19 |
| fungi | BObH_: okay, checking now | 19:19 |
| BObH_ | hansenb@us.ibm.com actually | 19:20 |
| openstackgerrit | Merged openstack-infra/infra-manual: Extend Gerrit ACL section https://review.openstack.org/181646 | 19:21 |
| *** matrohon has joined #openstack-infra | 19:21 | |
| *** sc68cal has joined #openstack-infra | 19:21 | |
| AJaeger | fungi, thanks ^ | 19:22 |
| *** ajmiller_ has joined #openstack-infra | 19:24 | |
| *** ajmiller has quit IRC | 19:24 | |
| fungi | BObH_: found it. anteaya, rloo and gordc requested we disable that account on may 5 because it was leaving machine-generated comments (and useless ones at that) on proposed changes such as https://review.openstack.org/178623 | 19:24 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for d-g git prune bug 1455123 https://review.openstack.org/183122 | 19:24 |
| openstack | bug 1455123 in OpenStack-Gate "fatal: unable to access 'https://git.openstack.org/openstack/sahara/': GnuTLS recv error (-9): A TLS packet with unexpected length was received." [Undecided,In progress] https://launchpad.net/bugs/1455123 - Assigned to Matt Riedemann (mriedem) | 19:24 |
| fungi | BObH_: "Bob Hansen - Patch Set 5: - Merge Failed. - This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset." | 19:24 |
| BObH_ | fungi, yes, that was definitely a mistake and has been corrected. | 19:24 |
| *** stevemar has quit IRC | 19:25 | |
| *** freerunner has quit IRC | 19:25 | |
| *** harlowja has quit IRC | 19:25 | |
| fungi | BObH_: okay, so it looks like you're running a zuul instance connected with that account. you've configured it to no longer comment on proposed changes for any projects? | 19:26 |
| *** harlowja has joined #openstack-infra | 19:26 | |
| BObH_ | fungi, yes, | 19:27 |
| fungi | BObH_: thanks. try logging in again; i've reactivated that account now | 19:27 |
| tim_o | we are experiencing a config issue with our test setup, so I've halted our neutron builds for the time being while we triage the issue. Is this the correct course of action? Just checking...thanks! | 19:28 |
| fungi | tim_o: if your system is reporting inaccurate results to proposed changes, then yes please disable it until you work out your configuration difficulties (otherwise someone is likely to ask us to disable the account for you) | 19:29 |
| *** HeOS has joined #openstack-infra | 19:29 | |
| BObH_ | fungi, thank you. I'm ok to use the sanbox until I get this baked? | 19:29 |
| openstackgerrit | Merged openstack-infra/project-config: Add notifications for all stable branches for kolla https://review.openstack.org/182133 | 19:30 |
| fungi | BObH_: sure, "sandbox" repositories are fair game | 19:30 |
| tim_o | thanks, fungi! | 19:30 |
| BObH_ | fungi, thanks again for your help | 19:30 |
| fungi | you're both welcome | 19:30 |
| *** ddutta has joined #openstack-infra | 19:31 | |
| ddutta | Hi @ajaeger | 19:32 |
| sdake | fungi ajeager thanks for the reviews on jeff's patch appreciate it :) | 19:33 |
| sdake | ajaeger that is :) | 19:33 |
| ddutta | hi .... have a new project proposal for review | 19:33 |
| ddutta | https://review.openstack.org/#/c/182886/ | 19:33 |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla https://review.openstack.org/182927 | 19:33 |
| *** weshay has quit IRC | 19:33 | |
| jpeeler | yes, hoepfully this is simple enough to get figured out quickly | 19:34 |
| *** alexpilotti has joined #openstack-infra | 19:35 | |
| fungi | sdake: jpeeler: sorry, i got distracted by random other requests and just now pushed my comments, but looks like AJaeger set you straight in the meantime | 19:35 |
| *** fawadkhaliq has quit IRC | 19:35 | |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla https://review.openstack.org/182927 | 19:36 |
| AJaeger | ddutta: as a stackforge project, you cannot publish on docs.openstack.org - your README needs adjustment at least | 19:36 |
| jpeeler | fungi: all good! feel free to take a look now if you can | 19:36 |
| AJaeger | The source is also wrong in the README | 19:36 |
| ddutta | ajaeger: thanks ... | 19:37 |
| ddutta | I just cloned with cookie cutter for this patch so I will just fix the docs right away | 19:37 |
| AJaeger | ddutta: other than that LGTM, +2 | 19:38 |
| ddutta | thanks ... | 19:38 |
| AJaeger | sdake, jpeeler looks fine - will +2A once jenkins is happy (which I expect) | 19:39 |
| ddutta | AJaegar: I have checked in my source repos and removed the docs | 19:40 |
| ddutta | thanks a lot for the quick review | 19:42 |
| *** angela-s has joined #openstack-infra | 19:45 | |
| *** ddutta has quit IRC | 19:48 | |
| *** shashankhegde has joined #openstack-infra | 19:48 | |
| *** cdent has joined #openstack-infra | 19:49 | |
| *** radez is now known as radez_g0n3 | 19:49 | |
| *** _nadya_ has joined #openstack-infra | 19:50 | |
| *** kmartin has quit IRC | 19:51 | |
| sdake | ddutta don't forget python-cognitiveclient ;-) | 19:52 |
| *** openstackgerrit has quit IRC | 19:52 | |
| *** openstackgerrit has joined #openstack-infra | 19:52 | |
| *** freerunner has joined #openstack-infra | 19:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Add image building verification to gate for Kolla https://review.openstack.org/182927 | 19:56 |
| *** rguillebert has joined #openstack-infra | 19:56 | |
| rguillebert | hi | 19:56 |
| rguillebert | I'm a pypy dev :) | 19:57 |
| *** cdent has quit IRC | 19:57 | |
| *** markvoelker has quit IRC | 19:57 | |
| rguillebert | I heard people need help for pypy support | 19:58 |
| openstackgerrit | Merged stackforge/python-jenkins: Add utf8 to headers https://review.openstack.org/165628 | 19:58 |
| *** dannywil_ has joined #openstack-infra | 19:58 | |
| fungi | hi rguillebert! sdague can probably explain the situation better than i, but in short openstack is lacking in a sufficient base of developers who want to spend time implementing and maintaining pypy support in our software | 19:59 |
| *** annegentle has joined #openstack-infra | 20:00 | |
| *** dannywilson has quit IRC | 20:00 | |
| fungi | including keeping an eye out for and triaging bugs and researching/reproducing/fixing test failures related to the same | 20:00 |
| fungi | alex gaynor did some earlier work on getting some of our software tested and working on pypy but hasn't been around for a while to continue keeping up with it | 20:02 |
| *** jtomasek has quit IRC | 20:02 | |
| rguillebert | that's bad :s | 20:02 |
| rguillebert | theoretically I can help out, but I don't have that much time | 20:03 |
| sdague | rguillebert: right, so right now our pypy jobs are just failing left right and center - http://status.openstack.org//elastic-recheck/#1290562 - 55 fails in 24 hrs / 497 fails in 10 days | 20:04 |
| sdague | rguillebert: so that's a concern, because that's how we got here in the first place. Without people that are committed to owning this, it's better to just remove it from our ci | 20:04 |
| rguillebert | yeah, that's a build config issue I think | 20:05 |
| sdague | yep, still an issue, and no owners | 20:06 |
| *** pc_m has quit IRC | 20:06 | |
| fungi | and our build configs are openly maintained by the community, just need people to step up and identify the causes and submit appropriate patches | 20:06 |
| fungi | in a continuous fashion of course, not just as a one-time fix | 20:07 |
| lifeless | rguillebert: hi! | 20:07 |
| rguillebert | hi :) | 20:07 |
| *** hdd has quit IRC | 20:07 | |
| lifeless | rguillebert: the social structure here - individual projects are happy to make their patches etc work with pypy as long as CI is running the jobs | 20:07 |
| lifeless | rguillebert: but when the jobs break systematically - e.g. the current failure - it usually need someone deeply familiar with quirks - e.g. 'must have setuptools X' or whatever to figure it out | 20:08 |
| lifeless | rguillebert: tl;dr weeks or months of no-effort-needed, then the occasional firedrill | 20:09 |
| fungi | and to work within the design limitations of the systems we use to perform this testing as well | 20:09 |
| rguillebert | well, I can be around and help with that sort of issue :) | 20:09 |
| *** asrangne has quit IRC | 20:09 | |
| rguillebert | as long as it's not a second full-time job :) | 20:09 |
| *** marun_ is now known as marun | 20:10 | |
| *** asrangne has joined #openstack-infra | 20:10 | |
| *** markvoelker has joined #openstack-infra | 20:11 | |
| rguillebert | does anyone know why the build uninstalls setuptools in the first place? | 20:12 |
| jpeeler | AJaeger: is there expected lag time for when the testing pipeline changes to just now merged review? | 20:13 |
| *** kgiusti has quit IRC | 20:14 | |
| rguillebert | and, can I run it locally? | 20:15 |
| lifeless | rguillebert: yeah, so there's a pile of context, let me give you a bootstrap | 20:18 |
| lifeless | rguillebert: all the CI jobs run in one-shot VM's | 20:18 |
| lifeless | rguillebert: there's automation to set those up, and they run from a custom base image (which is also automated) | 20:18 |
| *** mriedem has quit IRC | 20:18 | |
| lifeless | rguillebert: its entirely possible to run those locally (though not always simple) | 20:18 |
| lifeless | rguillebert: i'm not the best person to step you through that. | 20:19 |
| lifeless | rguillebert: also no, its not a full time job, not at all | 20:19 |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Add os-testr to global requirements list https://review.openstack.org/170268 | 20:19 |
| rguillebert | good :) | 20:19 |
| lifeless | rguillebert: uhm, why do we uninstall setuptools. I'm not sure we do. sdague ? fungi ? | 20:20 |
| *** Krinkle|detached is now known as Krinkle | 20:20 | |
| *** baoli has quit IRC | 20:21 | |
| fungi | lifeless: looks like as a workaround for bug 1290562 | 20:21 |
| openstack | bug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] https://launchpad.net/bugs/1290562 - Assigned to Ildiko Vancsa (ildiko-vancsa) | 20:21 |
| lifeless | k | 20:21 |
| lifeless | rguillebert: the top of https://bugs.launchpad.net/openstack-ci/+bug/1290562 shows setuptools being upgraded by pip | 20:21 |
| fungi | lifeless: specifically, many of the tox envs for pypy add a setuptools>3.4 | 20:21 |
| fungi | and then it looks like pip wants to uninstall the existing setuptools 2.2 in the virtualenv before replacing it with something later | 20:22 |
| lifeless | rguillebert: the reason it doesn't say 'running setup.py for setuptools' is because setuptools is a wheel | 20:22 |
| lifeless | rguillebert: and the pip ui around wheels is stil suboptimal | 20:22 |
| lifeless | fungi: deleting files before unpacking new ones is a common pip thing | 20:22 |
| lifeless | fungi: better than unpacking and then deleting :P | 20:22 |
| lifeless | s/unpacking/installing/ | 20:23 |
| openstackgerrit | John Griffith proposed openstack-infra/project-config: Add functional test job for cinderclient https://review.openstack.org/182528 | 20:23 |
| *** bhunter71 has quit IRC | 20:23 | |
| lifeless | clarkb: you said you think you've fixed the current thing? | 20:23 |
| lifeless | like, is there anything specific for rguillebert to do now, other than lurk here and highlight on pypy? | 20:23 |
| fungi | lifeless: too true. though for a while there was a situation where pip wouldn't immediately install the new thing after removing the old thing, thinking it could do it later on, and then run other things needing the deleted thing... mess | 20:23 |
| sdague | lifeless: it's definitely not fixed | 20:23 |
| lifeless | ok | 20:24 |
| sdague | by the ER graph | 20:24 |
| lifeless | so here's a recent one then | 20:24 |
| lifeless | http://logs.openstack.org/55/182755/3/gate/gate-requirements-pypy/242e044/console.html | 20:24 |
| fungi | lifeless: i think it's something similar, but not necessarily the same as what that bug report was working on | 20:24 |
| lifeless | http://logs.openstack.org/55/182755/3/gate/gate-requirements-pypy/242e044/console.html#_2015-05-14_20_05_07_830 | 20:24 |
| *** Sukhdev has joined #openstack-infra | 20:25 | |
| *** miqui has quit IRC | 20:25 | |
| fungi | because the workaround claims to be for a debian bug fixed in pypy-2.2.1+dfsg-2 | 20:25 |
| lifeless | yeah | 20:25 |
| lifeless | rguillebert: so ^ tht log | 20:25 |
| lifeless | rguillebert: shows that we're installing setuptools and the other test requirements | 20:25 |
| sdague | https://wiki.openstack.org/w/index.php?title=Special%3ASearch&profile=default&search=Liberty&fulltext=Search .... I hate media wiki search some times | 20:25 |
| *** Krinkle is now known as Krinkle|detached | 20:26 | |
| lifeless | then it gets down to the actual install step | 20:26 |
| rguillebert | yeah, looks like setuptools should be installed alone | 20:26 |
| lifeless | http://logs.openstack.org/55/182755/3/gate/gate-requirements-pypy/242e044/console.html#_2015-05-14_20_05_22_207 | 20:26 |
| sdague | anyone have the wiki for the etherpad list somewhere? | 20:26 |
| rguillebert | as in | 20:26 |
| rguillebert | update setuptools | 20:26 |
| fungi | oh! trusty-updates has 2.2.1+dfsg-1ubuntu0.2 so actually doesn't have that fix still | 20:26 |
| rguillebert | and then do everything else | 20:26 |
| rguillebert | because it looks like setuptools gets removed for future updating | 20:27 |
| lifeless | rguillebert: It looks like that, but the pip inner loop there is just horrid UI | 20:27 |
| lifeless | rguillebert: its replacing setuptools immediately: uninstall, install (from a wheel) | 20:27 |
| *** maishsk is now known as maishsk_afk | 20:27 | |
| *** mriedem has joined #openstack-infra | 20:27 | |
| lifeless | one thing we should do I think, to make this analysis easier, is to use pip list not pip freeze | 20:28 |
| lifeless | because pip list shows the versions of setuptools and pip | 20:28 |
| lifeless | freeze doesn't | 20:28 |
| rguillebert | 'install', '-U', 'setuptools>3.4', '-r/home/jenkins/workspace/gate-requirements-pypy/test-requirements.txt'] | 20:28 |
| *** soren_ has joined #openstack-infra | 20:28 | |
| *** matrohon has quit IRC | 20:28 | |
| rguillebert | that's what it's doing right? | 20:28 |
| lifeless | rguillebert: yeah | 20:28 |
| *** apmelton has quit IRC | 20:28 | |
| rguillebert | -r and beyond should be done separately | 20:29 |
| rguillebert | I think | 20:29 |
| *** kmartin has joined #openstack-infra | 20:29 | |
| fungi | as long as nothing else declares a conflicting install_requires on setuptools | 20:29 |
| reed | sdague, the list of etherpads for the design summit? | 20:30 |
| * reed hates mediawiki as a whole, not just its search | 20:31 | |
| reed | sdague, from https://wiki.openstack.org/wiki/Design_Summit/Liberty link to https://wiki.openstack.org/wiki/Design_Summit/Liberty/Etherpads | 20:31 |
| sdague | reed I found it | 20:31 |
| sdague | I was just amazingly surprised that Search "Liberty" only returned 1 page | 20:32 |
| sdague | which was completely unrelated to the summit | 20:32 |
| reed | intersting | 20:32 |
| lifeless | sdague: https://review.openstack.org/#/c/181777/ - replied to your comemnt | 20:32 |
| sdague | did the lucene worker die? | 20:32 |
| reed | i rely on firefox history to find wiki pages | 20:32 |
| lifeless | fungi: what pip version will be on that node? | 20:33 |
| reed | maybe the reboot yesterday left something borked? | 20:33 |
| sdague | so... if search is that bad, we should probably either delete that box entirely, or make it go to google site:wiki.openstack.org your thing | 20:33 |
| sdague | which is how I found it eventually | 20:33 |
| fungi | lifeless: in the global system environment, or the one bundled by virtualenv and used by tox? | 20:33 |
| sdague | the current model just induces anger | 20:33 |
| reed | search is not that bad actually | 20:33 |
| reed | let me try one thing | 20:33 |
| lifeless | fungi: the one being invoked in the failing pip execution | 20:34 |
| lifeless | fungi: its clearly not 6.1.1, so not current pip or current virtualenv | 20:34 |
| lifeless | rguillebert: ok, so I'm going to see if I can put together a patch to make that happen | 20:34 |
| reed | if I search "upstream" i have relevant results | 20:34 |
| lifeless | rguillebert: I'd like to know *why* though, because as I said pip isolated each actual install | 20:34 |
| reed | liberty not so much | 20:34 |
| lifeless | rguillebert: I know its not visible in the UI, but setuptools is actually installed after the 'uninstalling' line | 20:34 |
| *** weshay has joined #openstack-infra | 20:34 | |
| sdague | lifeless: so... I was pretty sure that was current behavior as of a couple of months ago | 20:35 |
| lifeless | rguillebert: wheel installs are invisible | 20:35 |
| reed | "outreachy" There were no results matching the query. | 20:35 |
| reed | impossible, something is broken on that search | 20:35 |
| lifeless | sdague: I don't know when it was fixed, but its fixed in 6.1.1 for sure, and up | 20:35 |
| lifeless | sdague: clarkb too was all paranoid and had to go manually try experiments to convince himself. | 20:35 |
| reed | should probably file a bug | 20:35 |
| reed | sdague, I have the impression that the search results are just stale | 20:36 |
| lifeless | sdague: but I have limmited care here. Its fixed upstream. If you don't believe it, please file a reproduction upstream, as it is an upstream issue. | 20:36 |
| fungi | lifeless: i wish tox logged that, even in the tox logfiles, but it looks like it does not. i'll jump on one of the machines and find out | 20:36 |
| reed | if I search "women" i get results to GnomeOutreachWomen | 20:36 |
| *** apmelton has joined #openstack-infra | 20:36 | |
| reed | which was renamed to Outreachy... but search Outreachy and get 0 | 20:36 |
| sdague | ok | 20:37 |
| sdague | lifeless: but if you are wrong later, you owes me a drink :) | 20:38 |
| reed | filing a bug | 20:38 |
| fungi | lifeless: currently for bare-trusty workers in hpcloud we seem to have virtualenv 1.11.4 installed, which comes with a bundled pip 1.5.4 | 20:38 |
| sdague | lifeless: +A on patch | 20:38 |
| lifeless | sdague: I'm happy to owe you a drink | 20:39 |
| lifeless | fungi: ok, so I think thats a huge problem :) | 20:39 |
| lifeless | cause 1.5.4 is frankly bad | 20:39 |
| vipul | question for folks here that may have used the Openstack ansible modules that are in ansible proper. Some of the modules seem really outdated (quantum) -- any recommendations of other / better implementations ? | 20:39 |
| fungi | lifeless: clarkb has merged a patch to bump that to latest virtualenv, but the last successful bare-trusty image update in hpcloud was 101.80 hours ago, before that patch merged | 20:39 |
| lifeless | fungi: rguillebert: I propose that we get latest virtualenv + pip + setuptools on the system as a prelude to running any tox commands | 20:39 |
| *** rossella_s has quit IRC | 20:40 | |
| lifeless | fungi: ok cool. | 20:40 |
| mordred | vipul hi | 20:40 |
| rguillebert | yeah that sounds good | 20:40 |
| vipul | wassup mordred | 20:40 |
| fungi | lifeless: this is why clarkb believes his patch has resolved the current pypy failure, we're just waiting for it to propagate to all our providers | 20:40 |
| mordred | vipul we're replacing them. I'll send you a link in 15 | 20:40 |
| *** otter768 has joined #openstack-infra | 20:40 | |
| fungi | lifeless: so i think this is probably good validation of that | 20:40 |
| lifeless | sdague: FWIW - the pip wheel loop is actually semantically wrong, because where we have open constraints, pip 8 will resolve a common set, which may not be latest, so we'd end up with the wrong wheels anyhow. | 20:41 |
| mordred | vipul you gonna be in Vancouver? | 20:41 |
| lifeless | fungi: alright | 20:41 |
| fungi | er, last successful bare-trusty image update in hpcloud-b5 anyway | 20:41 |
| lifeless | rguillebert: so - we're in hurry up and wait stage now, while automation does its thing. | 20:41 |
| vipul | mordred: yes sir.. | 20:41 |
| *** maishsk_afk has quit IRC | 20:41 | |
| fungi | which is where the example ran | 20:41 |
| lifeless | fungi: is that update pending, or broken? | 20:41 |
| vipul | mordred: i also looked at the shade stuff you showed a few months back.. is that still active? | 20:41 |
| mordred | cool. so, Monday afternoon session on the ansiblr modules | 20:41 |
| mordred | very much so | 20:42 |
| fungi | but looks like all the bare-trusty images in hpcloud aren't updating for similar duration now, so likely someone merged something 4-5 days ago which broke bare-trusty image builds | 20:42 |
| mordred | that's the basis of the new work ... it's what you want | 20:42 |
| fungi | i'll try to hunt that down | 20:42 |
| vipul | Ok great.. | 20:42 |
| vipul | mordred: Ansible collaboration day the topic? | 20:42 |
| clarkb | fungi or base image got renamed | 20:42 |
| sdague | lifeless: the patch has to recheck, because it last ran tests when we were enforcing pypy tests, but hopefully should land within a couple of hours | 20:42 |
| fungi | clarkb: also a distinct possibility | 20:42 |
| mordred | vipul yup! | 20:42 |
| vipul | ok i'll be there... | 20:43 |
| vipul | so shade is going into ansible proper i take it? | 20:43 |
| mordred | vipul morning is about deployment, afternoon about modules | 20:43 |
| mordred | the ansible mods are. shade is run through infra | 20:44 |
| mordred | nodepool is also moving to it | 20:44 |
| *** otter768 has quit IRC | 20:45 | |
| vipul | ok good to know. We'll start tinkering with it | 20:45 |
| mordred | woot! | 20:45 |
| lifeless | sdague: yeah, no rush | 20:46 |
| lifeless | sdague: I mean, its to let us hopefully test an alpha codebase :) | 20:46 |
| *** annegentle has quit IRC | 20:47 | |
| lifeless | rguillebert: thank you for stepping up; if you know another pypy fluent person that might be willing to do the same thing, we're all set. | 20:47 |
| sdague | heh, sure, but I like the idea of actually testing the pip upstream commits to solve our problem in the system that exposes said problem | 20:47 |
| fungi | also interesting, our nodepool image update logs stopped after the 11th, so i'm doing a manual update now to see what's going on there | 20:47 |
| jeblair | fungi: potential logging misconfiguration? | 20:48 |
| *** changbl has quit IRC | 20:48 | |
| fungi | jeblair: it's possible | 20:48 |
| jeblair | fungi: istr we landed some changes that touched logging | 20:48 |
| *** hdd has joined #openstack-infra | 20:48 | |
| *** dustins_ has quit IRC | 20:48 | |
| reed | https://storyboard.openstack.org/#!/story/2000265 not sure it should go to puppet-mediawiki though | 20:48 |
| rguillebert | lifeless, does that mean it's fixed? | 20:48 |
| reed | sdague, ^ | 20:48 |
| fungi | jeblair: just in hpcloud though. rax seems fine | 20:49 |
| *** dannywil_ has quit IRC | 20:49 | |
| lifeless | rguillebert: right now the proposed fix (new virtualenv) is stalled behind an unrelated problem | 20:49 |
| *** isviridov_away is now known as isviridov | 20:49 | |
| rguillebert | ok | 20:49 |
| lifeless | rguillebert: so we'll get that fixed, and then see if its fixed. If it is, \o/. If it isn't "ping rguillebert" :) | 20:49 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Rename stackforge/octavia -> openstack/octavia https://review.openstack.org/182748 | 20:49 |
| rguillebert | I'm available :) | 20:49 |
| lifeless | awesome | 20:49 |
| sdague | rguillebert: thanks! | 20:50 |
| openstackgerrit | Merged openstack/requirements: Uncap pytz and block pycadf 0.6.2 https://review.openstack.org/182755 | 20:51 |
| *** dannywilson has joined #openstack-infra | 20:51 | |
| *** Krinkle|detached is now known as Krinkle | 20:51 | |
| *** jaypipes has quit IRC | 20:53 | |
| *** _nadya_ has quit IRC | 20:53 | |
| reed | does OpenStack need to respond to Venom vulnerability? | 20:54 |
| *** Krinkle is now known as Krinkle|detached | 20:54 | |
| *** krtaylor has quit IRC | 20:55 | |
| fungi | reed: to what audience(s) are you thinking? | 20:55 |
| *** ianw has joined #openstack-infra | 20:56 | |
| *** bswartz has quit IRC | 20:56 | |
| *** dannywilson has quit IRC | 20:56 | |
| *** dboik_ has joined #openstack-infra | 20:56 | |
| reed | fungi, https://ask.openstack.org/en/question/66890/is-there-an-openstack-patch-for-the-venom-vuln-do-we-need-to-patch-qemu-manually/ | 20:56 |
| *** marun has quit IRC | 20:56 | |
| reed | I was about to write something there | 20:56 |
| *** baoli has joined #openstack-infra | 20:56 | |
| *** marun has joined #openstack-infra | 20:56 | |
| *** dprince has quit IRC | 20:57 | |
| sdake | fungi any chance you could approve this new project ajaeger already reviewed? | 20:58 |
| sdake | https://review.openstack.org/#/c/182886/ | 20:58 |
| *** dkranz has quit IRC | 20:59 | |
| *** dboik has quit IRC | 20:59 | |
| *** peristeri has quit IRC | 20:59 | |
| fungi | reed: probably someone could write an ossn similar to https://wiki.openstack.org/wiki/OSSN/OSSN-0012 but i'd suggest starting a [security] tagged thread on the dev ml referencing that (and maybe referencing the http://lists.openstack.org/pipermail/openstack-operators/2015-May/006945.html thread which is going on the ops ml) | 20:59 |
| *** dboik_ has quit IRC | 21:00 | |
| reed | ok | 21:01 |
| *** ildikov has quit IRC | 21:03 | |
| fungi | reed: but in short, venom is not a vulnerability in openstack software, it's a vulnerability in some software commonly used alongside openstack. we aren't the ones who provide fixes for it | 21:05 |
| *** ayoung has quit IRC | 21:06 | |
| fungi | fixes come from the xen project, possibly by way of distributions packaging the software | 21:06 |
| *** aysyanne has quit IRC | 21:06 | |
| *** gordc has quit IRC | 21:06 | |
| fungi | specifically in the form of patches to or patched versions of qemu | 21:07 |
| sdake | fungi thanks bro | 21:08 |
| *** chlong has quit IRC | 21:08 | |
| *** dizquierdo has joined #openstack-infra | 21:09 | |
| *** dstanek has quit IRC | 21:10 | |
| *** dannywilson has joined #openstack-infra | 21:10 | |
| jpeeler | fungi: is there expected lag time for when the testing pipeline changes to what has been merged in project-config? | 21:11 |
| *** rguillebert has quit IRC | 21:11 | |
| *** rguillebert has joined #openstack-infra | 21:11 | |
| fungi | jpeeler: generally no more than 45 minutes | 21:11 |
| *** ildikov has joined #openstack-infra | 21:13 | |
| jpeeler | yep, it has changed! | 21:13 |
| jpeeler | fungi: can sudo be used without issue? | 21:14 |
| openstackgerrit | Merged openstack-infra/project-config: Cognitive - Machine Learning as a Service https://review.openstack.org/182886 | 21:14 |
| openstackgerrit | Merged openstack-infra/project-config: Cognitive - Machine Learning as a Service https://review.openstack.org/182886 | 21:14 |
| *** dstanek has joined #openstack-infra | 21:14 | |
| *** markvoelker has quit IRC | 21:14 | |
| fungi | jpeeler: for the job you were updating, it looked like you weren't using the revoke-sudo builder macro, so it should be able to sudo on the worker to do whatever it wants | 21:14 |
| *** tiswanso has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:16 | |
| *** openstackgerrit has quit IRC | 21:22 | |
| *** openstackgerrit has joined #openstack-infra | 21:22 | |
| *** arielb has quit IRC | 21:23 | |
| *** rfolco has quit IRC | 21:24 | |
| *** markvoelker has joined #openstack-infra | 21:24 | |
| *** mwagner_lap has quit IRC | 21:25 | |
| clarkb | fungi: any luck with hpcloud trusty nodes? | 21:25 |
| *** tnovacik has quit IRC | 21:26 | |
| fungi | logging is busted, not sure why yet. my test build is still underway | 21:26 |
| fungi | it's already well into cloning the git cache | 21:26 |
| clarkb | cool | 21:27 |
| fungi | well, not cool. i'm sure it'll fail, it's just going to take plenty of time to do so apparently | 21:27 |
| clarkb | oh, but it sounds so promising :) | 21:27 |
| fungi | yeah, it's going to spend an hour getting my hopes up first before dashing them on the floor like so many broken dreams | 21:28 |
| *** annegentle has joined #openstack-infra | 21:28 | |
| *** fawadkhaliq has joined #openstack-infra | 21:28 | |
| *** Krinkle|detached is now known as Krinkle | 21:29 | |
| *** baoli has quit IRC | 21:29 | |
| *** jlanoux has quit IRC | 21:29 | |
| *** Sukhdev has quit IRC | 21:29 | |
| *** teran has joined #openstack-infra | 21:30 | |
| openstackgerrit | Debo~ Dutta proposed openstack-infra/project-config: Add python-cognitiveclient project for Cognitive https://review.openstack.org/183218 | 21:30 |
| *** Sukhdev has joined #openstack-infra | 21:30 | |
| *** yamahata has quit IRC | 21:32 | |
| *** tonytan4ever has quit IRC | 21:33 | |
| *** shashankhegde has quit IRC | 21:34 | |
| *** yfried is now known as yfried|afk | 21:35 | |
| clarkb | greghaynes: so I am reading the GitPython source and it seems to be shelling out to git to do clones so now I am really confused | 21:37 |
| clarkb | greghaynes: https://github.com/gitpython-developers/GitPython/blob/1.0.1/git/repo/base.py#L842 is what we call in zuul and that uses the Git object to run git clone | 21:37 |
| greghaynes | clarkb: hrm, you were mentioning something about an order of op's issue IIRC | 21:38 |
| greghaynes | (I havent fully paged in the context) | 21:38 |
| greghaynes | should be simple to make a testcase for zuul_cloner that repros I think | 21:38 |
| clarkb | greghaynes: well my original theory was that GitPython implemented git clone itself and did so poorly | 21:39 |
| *** e0ne has quit IRC | 21:39 | |
| clarkb | greghaynes: but it seems to fork a `git clone` to do the work so my theory is not really sound anymore | 21:39 |
| *** e0ne has joined #openstack-infra | 21:40 | |
| *** hashar has quit IRC | 21:40 | |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: kolla: add sudo for setup_docker.sh https://review.openstack.org/183226 | 21:40 |
| *** erlon has quit IRC | 21:41 | |
| fungi | i'm sort of sorry about this, but not really... <<<brace for impact>>> | 21:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/cookiecutter: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183229 | 21:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/specs-cookiecutter: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183231 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183233 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/elastic-recheck: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183234 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/gerritbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183235 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183236 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183237 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/jeepyb: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183238 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/jenkins-job-builder: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183239 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/nodepool: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183240 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/openstackid: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183241 | 21:42 |
| jeblair | oh dear | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183242 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-accessbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183243 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ansible: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183244 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-apparmor: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183245 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-askbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183246 | 21:42 |
| jeblair | some of our things have links don't they? | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-asterisk: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183247 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-bugdaystats: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183248 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-bup: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183249 | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-cgit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183250 | 21:42 |
| fungi | "some" yes | 21:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-diskimage_builder: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183251 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-drupal: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183252 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-elastic_recheck: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183253 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-elasticsearch: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183254 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-etherpad_lite: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183255 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-exim: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183256 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-gerrit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183257 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-gerritbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183258 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-github: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183259 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-graphite: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183260 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-haveged: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183261 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-httpd: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183262 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-iptables: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183263 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-jeepyb: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183264 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-jenkins: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183265 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-kerberos: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183266 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-kibana: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183267 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-lodgeit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183268 | 21:43 |
| sdague | jeblair: there are no links. The cake is a lie | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-log_processor: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183269 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-logrotate: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183270 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-logstash: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183271 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mailman: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183272 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183273 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-meetbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183274 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mysql_backup: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183275 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-nodepool: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183276 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openafs: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183277 | 21:44 |
| sdague | and fungi is about to win at stackalytics for the day | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackci: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183278 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackid: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183279 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-packagekit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183280 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-pip: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183281 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-planet: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183282 | 21:44 |
| *** mattfarina has quit IRC | 21:44 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-project_config: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183283 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-redis: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183284 | 21:44 |
| jeblair | we're all going to win on review counts | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-releasestatus: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183285 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-reviewday: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183286 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-simpleproxy: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183287 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-snmpd: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183288 | 21:44 |
| fungi | you'll catch up with me tomorrow, sdague | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ssh: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183289 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ssl_cert_check: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183290 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-statusbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183291 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-storyboard: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183292 | 21:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-subunit2sql: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183293 | 21:45 |
| *** shashankhegde has joined #openstack-infra | 21:45 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-sudoers: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183294 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-tmpreaper: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183295 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ulimit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183296 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-unattended_upgrades: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183297 | 21:45 |
| jeblair | fungi: i'm not sure about that -- it might be a while before anyone else gets a test node :) | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-unbound: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183298 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-user: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183299 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-zuul: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183300 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/pypi-mirror: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183301 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/shade: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183302 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183303 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard-webclient: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183304 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183305 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/tripleo-ci: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183306 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/yaml2ical: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183307 | 21:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183308 | 21:45 |
| mordred | oh my | 21:46 |
| fungi | it's fewer than 100 repos, at any rate. i'm sure we'll chew through the jobs | 21:46 |
| greghaynes | oh my indeed | 21:46 |
| *** yfried|afk is now known as yfried | 21:46 | |
| *** mriedem has quit IRC | 21:47 | |
| dougwig | whoa, someone ate their wheaties. | 21:47 |
| fungi | zuul was looking bored | 21:48 |
| sdague | yeh, it was at a pretty low job count | 21:48 |
| sdague | can't let it get out of shape | 21:48 |
| lifeless | fungi: what was the tl;dr on the image? | 21:48 |
| *** ociuhandu has quit IRC | 21:48 | |
| fungi | lifeless: my test update just finished failing and i'm looking through the log now | 21:49 |
| *** spzala has quit IRC | 21:49 | |
| fungi | ClientException: Unknown Error (HTTP 500) | 21:50 |
| fungi | bubbling up from novaclient | 21:50 |
| mtreinish | fungi: shooting for #1 in liberty, I have faith you can take the top spot this cycle | 21:50 |
| openstackgerrit | Merged openstack-dev/cookiecutter: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183229 | 21:50 |
| fungi | this is at the point is should have tried to make a snapshot | 21:50 |
| fungi | did we maybe have something change around the snapshot method (either in nodepool or in novaclient)? | 21:51 |
| fungi | the actual template system seems to puppet successfully | 21:51 |
| *** weshay has quit IRC | 21:52 | |
| jeblair | this is going throw my +/- ratio out of whack | 21:53 |
| mordred | jeblair: :) | 21:53 |
| fungi | http://paste.openstack.org/show/223168 | 21:53 |
| fungi | could that maybe be a quota problem in hpcloud? | 21:54 |
| fungi | like we have too many snapshots or something? | 21:54 |
| nibalizer | we're seeing hpcloud errors downstream so... | 21:54 |
| jeblair | fungi: i hope you left one error in the stack to make sure we're really reviewing it | 21:54 |
| mordred | jeblair: hehe | 21:55 |
| mordred | jeblair: you know what - I AM actually looking at the darned things - because I'm a moron | 21:55 |
| clarkb | greghaynes: did we hit the git thing on trusty? | 21:55 |
| *** dkranz has joined #openstack-infra | 21:55 | |
| jeblair | mordred: i am too. i mean, i'm putting my name on it. | 21:55 |
| fungi | jeblair: i suspect there may be an error or two. i'm mostly going back through the non openstack-infra ones i proposed updates to because those are less regular | 21:55 |
| clarkb | greghaynes: I just cobbled together a thing and it seems to be ok so far | 21:55 |
| jeblair | fungi: i'm saving suggestions for improvement for a later time | 21:56 |
| greghaynes | clarkb: pretty sure yes, I want to say all the dib tests were failing | 21:56 |
| mordred | jeblair: yah. still ... I feel mildly silly anyway | 21:56 |
| clarkb | fungi: we did switch to shade but its using the openstack clients from shade so not much really changed | 21:56 |
| clarkb | fungi: also I think that was less than 100 hours ago | 21:56 |
| greghaynes | yea, if you look at that traceback its not even hitting shade | 21:57 |
| *** yfried is now known as yfried|afk | 21:57 | |
| *** hdd has quit IRC | 21:57 | |
| *** jgrimm is now known as zz_jgrimm | 21:57 | |
| greghaynes | the big change was really using os-client-config | 21:57 |
| fungi | oh! here's a possibility... could the cli not be in sync with the daemon for image update? | 21:57 |
| kambiz | does anyone know if there's a way to get ops in a channel on freenode, if the channel is not registered and there are no ops in the channel ? | 21:57 |
| mordred | aroo? | 21:57 |
| mordred | fungi: whatcho mean? | 21:58 |
| jeblair | fungi: found a -1. ;) | 21:58 |
| mordred | oh - well... _installing_ shade will have upgraded theclient libs | 21:58 |
| mordred | jeblair: nice catch! | 21:58 |
| fungi | jeblair: awesome! | 21:58 |
| fungi | mordred: in the past we've had at least one instance of the nodepool cli was calling into a different code path than the daemon was using to do some functions | 21:58 |
| clarkb | fungi: no they should be the same since we rebooted | 21:59 |
| mordred | jeblair, fungi: shade has docs - should the homepage be those and not the system-config? | 21:59 |
| fungi | mordred: probably | 21:59 |
| clarkb | greghaynes: more progress, things get weird after doing the repo.reset() in the cloner | 21:59 |
| clarkb | greghaynes: before that it actually looks pretty sane | 21:59 |
| mordred | if so - I found a -1 to! | 21:59 |
| mordred | yay! | 21:59 |
| jeblair | mordred: yeah, though that was part of the 'improvements' that i was leaving for later | 21:59 |
| jeblair | mordred: that's true for several of those | 21:59 |
| mordred | ah | 21:59 |
| mordred | k | 21:59 |
| mordred | well, I'm keeping the -1 since I left it anyway :) | 22:00 |
| fungi | mordred: though the homepage on it went to the system-config docs previously (we didn't have the "right" url for it before either) | 22:00 |
| jeblair | i'll +2 it either way. mordred and fungi can fight it out :) | 22:00 |
| jeblair | mordred: you should +2 fungis and then write a bunch of patches to improve the links ;) | 22:00 |
| fungi | or feel free to hijack mine | 22:00 |
| *** dkranz has quit IRC | 22:01 | |
| fungi | i won't mind (or likely even notice) | 22:01 |
| *** e0ne has quit IRC | 22:01 | |
| *** dkranz_ has joined #openstack-infra | 22:01 | |
| *** teran_ has joined #openstack-infra | 22:01 | |
| fungi | so... yeah this is calling directly from the provider manager into novaclient | 22:02 |
| *** Rockyg has joined #openstack-infra | 22:02 | |
| openstackgerrit | Merged openstack-infra/puppet-apparmor: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183245 | 22:02 |
| fungi | trying to do a createImage call | 22:02 |
| jeblair | mordred: you missed 183239 | 22:02 |
| mordred | jeblair: I may not subscribe to that repo | 22:02 |
| clarkb | greghaynes: so flipping the reset and prune order in the cloner works | 22:02 |
| mordred | jeblair: repo? | 22:02 |
| clarkb | jeblair: ^ any idea why that would fix things? | 22:02 |
| jeblair | mordred: jjb | 22:03 |
| mordred | jeblair: yeah - I don't gertty jjb | 22:03 |
| clarkb | I need to read up on prune | 22:03 |
| greghaynes | clarkb: prune docs make it sound like it should be safe... | 22:03 |
| jeblair | clarkb: help me context switch please | 22:03 |
| *** teran has quit IRC | 22:04 | |
| clarkb | jeblair: the zuul-cloner breakage that greghaynes ran into with diskimage-buidler on our dib built test nodes | 22:04 |
| *** markvoelker has quit IRC | 22:04 | |
| jeblair | mordred: well, you can just ctrl-o "183239" and review it :) | 22:04 |
| clarkb | jeblair: git.exc.GitCommandError: 'git branch -d -r origin/fetch_master' returned with exit code 128 returns stderr: 'fatal: HEAD not found below refs/heads!' | 22:04 |
| clarkb | jeblair: that happens if we git clone, git reset, then git prune. It does not happen if we git clone, git prune, git reset | 22:04 |
| clarkb | my reading of this is that order shouldn't matter and yet.. | 22:04 |
| greghaynes | clarkb: I wonder if the prune is just surfacing the bug, and its actually broken reguardless | 22:05 |
| *** SotK has joined #openstack-infra | 22:05 | |
| jeblair | clarkb: ok. iirc, we decided we should (a) make dib not leave weird things in the repo and (b) make zuul-cloner more resilient if weird things are left in the repo | 22:05 |
| greghaynes | jeblair: yep, this is b | 22:05 |
| clarkb | correct | 22:05 |
| *** bhuvan has quit IRC | 22:05 | |
| clarkb | prune relies on https://github.com/gitpython-developers/GitPython/blob/e9f8f159ebad405b2c08aa75f735146bb8e216ef/git/remote.py#L449 | 22:05 |
| clarkb | and now I think I know the issue | 22:05 |
| clarkb | you have to make a new repo object each time because lol | 22:06 |
| mordred | jeblair: ctrl-o did not work for me | 22:06 |
| clarkb | I will confirm and write a patch | 22:06 |
| *** isviridov is now known as isviridov_away | 22:06 | |
| jeblair | mordred: what happened? | 22:06 |
| mordred | jeblair: it said "change not found" | 22:06 |
| jeblair | clarkb: cute | 22:06 |
| openstackgerrit | James Carey proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks https://review.openstack.org/134052 | 22:06 |
| clarkb | hrm except zuul is doing that for us | 22:06 |
| jeblair | mordred: try again? | 22:06 |
| jeblair | mordred: and if you have debug logs from that, i'd love a copy | 22:07 |
| mordred | jeblair: now it worked | 22:07 |
| mordred | jeblair: and no - I didn't start with debug this time | 22:07 |
| mordred | sorry | 22:07 |
| jeblair | mordred: any reason you don't always run with debug? | 22:07 |
| jeblair | mordred: it will really help me fix bugs for you :) | 22:07 |
| fungi | mordred: jeblair: for 183305 those urls exist and are the counterparts to the original links. is the concern that they were the wrong links originally and we should point to the application manuals for those rather than their pages in our system configuration manuals? | 22:08 |
| mordred | jeblair: I'll try to remember that in the future | 22:08 |
| *** doug-fish has left #openstack-infra | 22:08 | |
| mordred | jeblair: you know, you could have it email you tracebacks ... | 22:08 |
| jeblair | fungi: oh, did i get that backwards, sorry. | 22:08 |
| jeblair | fungi: i was wrong; review updated. | 22:09 |
| mordred | clarkb: for lxml - shouldn't we ensure => absent for lxml? | 22:09 |
| clarkb | mordred: no because zuul still relies on that package | 22:09 |
| clarkb | mordred: its a avoid trouble with puppet problem | 22:10 |
| mordred | clarkb: but you removed the sentry | 22:10 |
| mordred | entry | 22:10 |
| clarkb | mordred: but it will conflict with the entry zuul uses | 22:10 |
| jeblair | mordred: i just double checked by opening a random nova change, and it worked. :/ | 22:10 |
| mordred | clarkb: this is on puppet-nodepool | 22:10 |
| clarkb | mordred: yes, I know | 22:10 |
| mordred | ok. that seems strange to me and like evil is happening | 22:11 |
| clarkb | mordred: the idea is that zuul and nodepool are coninstallable, which is why we guarded the python-lxml install previously | 22:11 |
| mordred | yah | 22:11 |
| clarkb | mordred: but that means we cannot ensure present in one and ensure absent in the other | 22:11 |
| mordred | so - shouldn't we need this same thing for zuul | 22:11 |
| mordred | ? | 22:11 |
| clarkb | puppet will explode | 22:11 |
| *** bhunter71 has joined #openstack-infra | 22:11 | |
| fungi | i'm getting the feeling that this createImage 500 error from the api in hpcloud is causing nodepool not to actually write out the corresponding image logs there | 22:11 |
| clarkb | mordred: no because pip generally figures it out for us | 22:11 |
| mordred | if pip install -U is causing carnage on nodepool it's going to on zuul too | 22:11 |
| *** Swami has quit IRC | 22:11 | |
| jeblair | fungi: logging should be continuous | 22:11 |
| clarkb | mordred: no it won't because no shade on zuul | 22:11 |
| fungi | hrm, so not buffered until complete? | 22:11 |
| jeblair | fungi: correct | 22:11 |
| clarkb | mordred: the problem is shade specifically installs keystoneclient. | 22:12 |
| fungi | rereading | 22:12 |
| mordred | clarkb: what depends on lxml? | 22:12 |
| jeblair | fungi: unless something changed | 22:12 |
| clarkb | mordred: so right now on zuul we don't have to go down that upgrade path and its fine | 22:12 |
| clarkb | mordred: it will only cause carnage if we update things (like we did with shade) such that pip decides it should upgrade lxml | 22:12 |
| mordred | right. but if the two are supposed to be co-instalalble | 22:13 |
| clarkb | mordred: remember pip dep resolution doesn't look at the whole tree at once, so if you hide lxml behind something that does not need an upgrade you are fine | 22:13 |
| mordred | then shade will carnage the zuul install if they co-nstall | 22:13 |
| mordred | and we know it | 22:13 |
| *** dizquierdo has quit IRC | 22:13 | |
| clarkb | mordred: no zuul will just get latest lxml via nodepool | 22:13 |
| clarkb | mordred: it will work fine | 22:13 |
| mordred | except it will still have python-lxml instaled | 22:13 |
| mordred | ? | 22:13 |
| clarkb | mordred: yes | 22:13 |
| *** spzala has joined #openstack-infra | 22:13 | |
| mordred | seems half-done to me and I'm pretty sure it's going to break someone | 22:14 |
| mordred | because I do not bleieve pip will do the right thing with pyhton-lxml installed ia pacakges | 22:14 |
| mordred | I might be wrong - but it seems like playing with fire | 22:14 |
| *** D1337d has joined #openstack-infra | 22:15 | |
| mordred | fungi: HP cloud is doing maint for the thing | 22:15 |
| mordred | fungi: so our fail there may be related | 22:15 |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: kolla: add sudo for setup_docker.sh https://review.openstack.org/183226 | 22:15 |
| clarkb | mordred: two things, first pip installs to /usr/local and thats earlier in python path so that bit should just work. Without puppet package sanity I can't think of a better solution and still preserve coinstallability | 22:16 |
| fungi | mordred: oh... maybe the real problem is shadowed by venom maintenance | 22:16 |
| fungi | got it | 22:16 |
| clarkb | mordred: we could force zuul to do the unnecessary build lxml from source | 22:16 |
| *** eharney has quit IRC | 22:17 | |
| *** sdake has quit IRC | 22:17 | |
| *** ddieterly has quit IRC | 22:17 | |
| mordred | clarkb: I just remember carnage when we've had competing system and python things ... it may be fixed now - but I haven't proven to myself that it's fixed | 22:18 |
| *** sdake has joined #openstack-infra | 22:18 | |
| clarkb | mordred: if we do do that we would haev to make this as an intermediate step anyways before ensuring absent otherwise puppet will break | 22:18 |
| mordred | clarkb: although your words make sense and is how I'd like to have it work | 22:18 |
| clarkb | mordred: what kind of carnage? I have never seen it reak | 22:18 |
| clarkb | mordred: the only thing similar to carnage I have seen is if yu upgrade pip during a puppet run pip provider in puppet gets confused about where to get pip in its path | 22:18 |
| clarkb | mordred: but thats mostly a puppet process caching things problem | 22:18 |
| *** bknudson has quit IRC | 22:20 | |
| *** spzala has quit IRC | 22:20 | |
| *** nelsnelson has quit IRC | 22:20 | |
| jpeeler | sorry to nag, but can i get a quick review on https://review.openstack.org/#/c/183195/ ? | 22:20 |
| *** annegentle has quit IRC | 22:20 | |
| *** ddieterly has joined #openstack-infra | 22:21 | |
| *** ivar-lazzaro has quit IRC | 22:21 | |
| *** ayoung has joined #openstack-infra | 22:22 | |
| *** ivar-lazzaro has joined #openstack-infra | 22:24 | |
| *** markvoelker has joined #openstack-infra | 22:25 | |
| *** rlandy has quit IRC | 22:25 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183236 | 22:25 |
| openstackgerrit | Merged openstack-infra/gerritbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183235 | 22:26 |
| openstackgerrit | Merged openstack/requirements: Allow testing with specific pips https://review.openstack.org/181777 | 22:26 |
| openstackgerrit | Merged openstack-infra/infra-specs: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183237 | 22:27 |
| *** ildikov has quit IRC | 22:28 | |
| clarkb | jeblair: new thing I am noticing about zuul-cloner is that it kills refs/heads/HEAD somewhere | 22:28 |
| *** ivar-lazzaro has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack-infra/jeepyb: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183238 | 22:29 |
| *** ildikov has joined #openstack-infra | 22:29 | |
| clarkb | oh wait, no its complaining that the root HEAD is not defined in refs/HEADS | 22:30 |
| clarkb | er refs/heads | 22:30 |
| *** ivar-lazzaro has joined #openstack-infra | 22:30 | |
| clarkb | which should be ok if I do the full cloen I think | 22:30 |
| clarkb | its just in an unhappy state halfway through our edits | 22:31 |
| clarkb | greghaynes: jeblair I have figured it out | 22:33 |
| greghaynes | oh/ | 22:33 |
| clarkb | will patch with a lengthy commit message | 22:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Add priority efforts section https://review.openstack.org/183337 | 22:33 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183234 | 22:34 |
| openstackgerrit | Merged openstack-infra/nodepool: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183240 | 22:34 |
| clarkb | ya, so when we do a reset HEAD points to refs/remotes/origin/master it does not point to refs/heads/SOMETHING. git branch explodes in this case which is the error message we saw: stderr: 'fatal: HEAD not found below refs/heads!' | 22:34 |
| clarkb | so we can fix this by pruning first then reseting so that git branch is not run until later after we have done a checkout of the final desired state | 22:34 |
| * SpamapS uploads python-shade to Debian unstable | 22:35 | |
| SpamapS | zigo: hey, we need to talk about adding python3 packages to the openstack clients. :) | 22:35 |
| clarkb | greghaynes: jeblair so this really has little to do with dib and is a zuul-clner bug, we just happen to only tickle it if prune is not a noop | 22:36 |
| zigo | SpamapS: I'd like to do so, but before this happens, we need Python3 support for pysaml2 !!! | 22:36 |
| zigo | SpamapS: That's currently the only remaining blocker. | 22:37 |
| zigo | SpamapS: It's needed to fix python-keystoneclient. | 22:37 |
| zigo | Then when this is done, I can add it to all packages. | 22:37 |
| zigo | SpamapS: Would you help? | 22:37 |
| SpamapS | zigo: patches awaiting merge, or more? | 22:37 |
| zigo | SpamapS: *huge* work to be done on this upstream. | 22:37 |
| SpamapS | zigo: yeah I can always find a little time for python3 :) | 22:37 |
| zigo | SpamapS: I started to write it, and the patch I wrote ended up being like 5000 long. | 22:38 |
| zigo | SpamapS: Then I did some crap which broke stuff, and gave up... | 22:38 |
| zigo | SpamapS: I'd need to restart it. | 22:38 |
| *** teran has joined #openstack-infra | 22:38 | |
| greghaynes | clarkb: yea, the dib thing definitely isnt a 'bug', just silly | 22:38 |
| *** soren_ has quit IRC | 22:38 | |
| clarkb | greghaynes: ya its weird enough that we never hit this corner before :) | 22:39 |
| *** teran_ has quit IRC | 22:39 | |
| clarkb | anyways I should have a patch up in a bit, want to make sure I grok this before writing the commit message that justifies changing the order | 22:39 |
| clarkb | but changing the order does fix | 22:39 |
| zigo | SpamapS: Would you try? | 22:39 |
| SpamapS | zigo: poking now | 22:39 |
| openstackgerrit | Merged openstack-infra/infra-manual: Add ChangeLog and AUTHORS to .gitignore https://review.openstack.org/182878 | 22:39 |
| zigo | SpamapS: I can send you my patch file. | 22:40 |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: kolla: add sudo for setup_docker.sh https://review.openstack.org/183226 | 22:40 |
| clarkb | SpamapS: zigo I wonder if the keystone auth lib split out thing will remove need for pysaml2 in py3k for the base use case? | 22:40 |
| clarkb | SpamapS: zigo might be worth finding out from morganfainberg or jamielennox|away as that may speed things along with less effort | 22:41 |
| openstackgerrit | Merged openstack-infra/puppet-bugdaystats: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183248 | 22:41 |
| openstackgerrit | Merged openstack-infra/project-config: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183242 | 22:41 |
| *** marun has quit IRC | 22:41 | |
| SpamapS | clarkb: no, the thing everybody needs is the thing that needs saml. | 22:41 |
| sdake | could a couple cores fix our busted gate by reviewing: https://review.openstack.org/#/c/183226/ | 22:41 |
| zigo | clarkb: As much as I know, the keystone_authtoken thing is *planned* to move out from keystoneclient to the middleware, but that's not done yet. | 22:41 |
| openstackgerrit | Merged openstack-infra/puppet-github: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183259 | 22:41 |
| *** otter768 has joined #openstack-infra | 22:41 | |
| zigo | clarkb: We'd better just fix pysaml2 anyway... | 22:42 |
| sdake | thanks jeblair | 22:42 |
| clarkb | SpamapS really? I dont think we use it | 22:43 |
| openstackgerrit | Merged openstack-infra/puppet-ansible: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183244 | 22:43 |
| openstackgerrit | Merged openstack-infra/puppet-elasticsearch: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183254 | 22:43 |
| SpamapS | clarkb: hm, actually, now that I read deeper, it's just a plugin. | 22:43 |
| morganfainberg | zigo: auth token has been removed from ksc | 22:43 |
| openstackgerrit | Merged openstack-infra/puppet-askbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183246 | 22:43 |
| SpamapS | zigo: ^ | 22:43 |
| openstackgerrit | Merged openstack-infra/puppet-drupal: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183252 | 22:43 |
| mordred | zigo, SpamapS: also - keystoneauth has been release the first version | 22:43 |
| zigo | Cool. | 22:43 |
| morganfainberg | zigo: we are waiting post summit for a 2.x release of ksc because we don't want to be fixing broken things at the summit. | 22:44 |
| zigo | mordred: keystoneauth ? As a separate package? | 22:44 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183253 | 22:44 |
| openstackgerrit | Merged openstack-infra/puppet-asterisk: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183247 | 22:44 |
| SpamapS | so yeah, I don't even see pysaml2 in the reqs for master keystoneclient | 22:44 |
| morganfainberg | Keystoneauth has been split which is just session object etc. | 22:44 |
| mordred | zigo: yah - intended to be used by other libs to auth against keystone instead of keystoneclient | 22:44 |
| morganfainberg | And we have a separate package just for saml auth plugin in ksc | 22:44 |
| mordred | so that keystoneclientis about talking to the keystone rest api | 22:44 |
| morganfainberg | So the base one doesn't need py saml | 22:44 |
| mordred | ++ | 22:44 |
| zigo | mordred: Wonderful. | 22:44 |
| mordred | it's very exciting | 22:44 |
| SpamapS | morganfainberg: but keystoneauth will still need pysaml2 for clouds that use saml yes? | 22:44 |
| zigo | SpamapS: Then we need all clients to switch to keystoneauth instead of ksc. | 22:45 |
| *** annegentle has joined #openstack-infra | 22:45 | |
| mordred | SpamapS: only the keystoneauth plugin for saml | 22:45 |
| mordred | which will be different | 22:45 |
| morganfainberg | SpamapS: only if you want the plugin | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-exim: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183256 | 22:45 |
| mordred | keystoneauth is puggable | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-accessbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183243 | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183255 | 22:45 |
| morganfainberg | And that is in a separate package. | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-cgit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183250 | 22:45 |
| SpamapS | Right, so can just make that its own package that has no python3 equiv until pysaml2 is ported. | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-gerritbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183258 | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-bup: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183249 | 22:45 |
| mordred | ++ | 22:45 |
| morganfainberg | Yep | 22:45 |
| mordred | it's almost like we're doing this right :) | 22:45 |
| openstackgerrit | Merged openstack-infra/puppet-kerberos: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183266 | 22:45 |
| *** otter768 has quit IRC | 22:46 | |
| morganfainberg | Keystoneauth has 0.1.0 out today. Please file bugs (Lp) or complaints for what we need to fix before we move to 1.x which will be the stable release we start from. | 22:46 |
| morganfainberg | Our target for 1.x is around milestone 1 | 22:46 |
| sdake | thanks fungi | 22:46 |
| morganfainberg | To add to g-r so we can get everyone using it where appropriate in Liberty. | 22:46 |
| zigo | Gosh, how come there's argparse in requirements.txt of keystoneauth ... :( | 22:47 |
| morganfainberg | mordred: phsaw doing it right or doing it in a way that people stop complaining at me ;). | 22:47 |
| zigo | Let's get rid of these, PLEASE ! | 22:47 |
| clarkb | zigo: because python2.6 | 22:47 |
| clarkb | zigo: but its fine | 22:47 |
| *** annegentle has quit IRC | 22:47 | |
| zigo | clarkb: I thought we're not supporting it anymore... | 22:48 |
| morganfainberg | zigo: it isn't done. We just did the split most requirements will get removed as we continue cleanup. | 22:48 |
| *** ajo has quit IRC | 22:48 | |
| clarkb | zigo: the libs still support it until the last server release that supports it goes away | 22:48 |
| morganfainberg | Keystoneauth needs to work with 2.6 until clients don't need to. | 22:48 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183257 | 22:48 |
| openstackgerrit | Merged openstack-infra/puppet-graphite: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183260 | 22:48 |
| morganfainberg | If a client needs 2.6, ksa needs 2.6 | 22:48 |
| clarkb | morganfainberg: right | 22:49 |
| morganfainberg | But we do plan on eliminating Oslo.serialization from the requirements and a few others. | 22:49 |
| morganfainberg | Before 1.x release. | 22:49 |
| SpamapS | mordred: /win 20 | 22:50 |
| SpamapS | doh | 22:50 |
| SpamapS | mordred: forgot to ^5 you, then go to window 20 | 22:50 |
| *** Swami has joined #openstack-infra | 22:50 | |
| openstackgerrit | Merged openstack-infra/storyboard: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183303 | 22:51 |
| openstackgerrit | Merged openstack-infra/puppet-httpd: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183262 | 22:51 |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183264 | 22:51 |
| zigo | SpamapS: Ok, so to go back to reality: if we get a patch for pysaml2 to support Py3, then the next day, I can start adding Py3 support to client packages. | 22:51 |
| zigo | (ie: it wont need to wait until we do a major work everywhere...) | 22:51 |
| zigo | So, in other words, it seems more realistic to just port pysaml2 to Py3 now. | 22:51 |
| zigo | The other Py3 blocker for packages is qpid (impacting oslo.messaging and it's dependencies), though we had a thread about it on the list, so I guess you know already. | 22:52 |
| mordred | drop qpid | 22:52 |
| SpamapS | zigo: has to be done anyway | 22:52 |
| zigo | Agreed. | 22:52 |
| SpamapS | zigo: you could just make the drivers separate packages. :) | 22:53 |
| morganfainberg | SpamapS: ++ | 22:53 |
| SpamapS | qpid will be deprecated in Liberty, but not removed until "M" | 22:53 |
| SpamapS | so I'd say do that anyway | 22:54 |
| SpamapS | python-oslo-messaging-qpid or something. | 22:54 |
| *** hemna is now known as hemnafk | 22:55 | |
| SpamapS | ugh | 22:55 |
| SpamapS | pysaml2's test suite requires mongodb | 22:55 |
| *** banix has joined #openstack-infra | 22:55 | |
| * SpamapS stabs | 22:55 | |
| jhesketh | Morning | 22:55 |
| mtreinish | SpamapS: heh, so it's webscale testing :) | 22:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul: Fix git prune order in zuul-cloner https://review.openstack.org/183348 | 22:57 |
| clarkb | greghaynes: jeblair jhesketh ^ | 22:57 |
| SpamapS | mtreinish: should use the /dev/null driver instead | 22:57 |
| openstackgerrit | Jeff Peeler proposed openstack-infra/project-config: make kolla jobs non-voting https://review.openstack.org/183349 | 22:57 |
| *** markvoelker has quit IRC | 22:58 | |
| morganfainberg | SpamapS: it's ok, I'm going to make Keystoneauth require mongodb but only for the release you work with *shiftyeyes* | 22:59 |
| *** sabeen1 has quit IRC | 22:59 | |
| *** ociuhandu has joined #openstack-infra | 22:59 | |
| greghaynes | clarkb: fun. lgtm | 22:59 |
| mordred | clarkb: lg | 23:00 |
| *** ashleighfarnham has quit IRC | 23:00 | |
| *** pabelanger has quit IRC | 23:01 | |
| jhesketh | clarkb: lgtm, approved | 23:01 |
| *** rcarrillocruz has quit IRC | 23:01 | |
| *** bswartz has joined #openstack-infra | 23:01 | |
| clarkb | awesome, ty | 23:02 |
| SpamapS | morganfainberg: :| | 23:04 |
| *** redrobot has quit IRC | 23:04 | |
| clarkb | mordred: 182895 is another recent "fix this thing" change | 23:04 |
| *** ayoung has quit IRC | 23:04 | |
| *** Trozz has quit IRC | 23:04 | |
| clarkb | jhesketh: ^ not sure how interested you are in log cleanup :) | 23:04 |
| *** markvoelker has joined #openstack-infra | 23:05 | |
| *** redrobot has joined #openstack-infra | 23:05 | |
| *** redrobot is now known as Guest19895 | 23:05 | |
| *** SamYaple has quit IRC | 23:05 | |
| *** freerunner has quit IRC | 23:06 | |
| jhesketh | clarkb: I'm interested in anything that helps infra :-). | 23:07 |
| *** ashleighfarnham has joined #openstack-infra | 23:07 | |
| pleia2 | jhesketh: ^5 | 23:07 |
| *** ddieterly has quit IRC | 23:07 | |
| clarkb | woot | 23:08 |
| *** davideagnello has quit IRC | 23:08 | |
| jhesketh | clarkb: doesn't elasticsearch go back 10 days or something? Should we clean out everything older than 14 days instead? | 23:08 |
| *** asrangne__ has joined #openstack-infra | 23:08 | |
| clarkb | jhesketh: the indexed logs inside of elasticsearch go back 10 days. The logs we are cleaning up here are the normal text file logs for the elasticsearch service itself | 23:09 |
| clarkb | jhesketh: basically its turtles, but good suggestion since we may need to know why $thing that is indexed was indexed | 23:09 |
| SpamapS | zigo: whats the story on python-memcached ? Looks like it may be dead? | 23:09 |
| clarkb | jhesketh: I can update to 14 days | 23:09 |
| jhesketh | Right, yes that was my point | 23:09 |
| jhesketh | I think it likely won't matter but 14 days makes more sense | 23:10 |
| *** chlong has joined #openstack-infra | 23:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-elasticsearch: Cleanup old ES logs https://review.openstack.org/182895 | 23:10 |
| clarkb | jhesketh: yup ^ its a sane idea | 23:10 |
| *** asrangne has quit IRC | 23:11 | |
| jhesketh | clarkb: thanks, lgtm | 23:11 |
| *** davideagnello has joined #openstack-infra | 23:11 | |
| jeblair | clarkb: zuul-cloner change makes sense. thanks | 23:11 |
| clarkb | jeblair: it ended up being far more time to debug than fix :) | 23:13 |
| clarkb | those are the best bugs | 23:13 |
| *** panda has quit IRC | 23:13 | |
| *** dstufft has quit IRC | 23:14 | |
| *** bhuvan has joined #openstack-infra | 23:14 | |
| *** panda has joined #openstack-infra | 23:14 | |
| *** dangers is now known as dangers_away | 23:14 | |
| *** whoops has quit IRC | 23:14 | |
| *** teran_ has joined #openstack-infra | 23:14 | |
| *** salv-orlando has joined #openstack-infra | 23:15 | |
| *** dstufft has joined #openstack-infra | 23:15 | |
| *** atan8 has quit IRC | 23:16 | |
| *** jamespage has quit IRC | 23:17 | |
| *** gnuoy has quit IRC | 23:17 | |
| *** teran has quit IRC | 23:17 | |
| jlvillal | Is there a way in Zuul or Jenkins to see the results of previous partial test runs? Test ran and finished some of the tests, but then before everything was done a new patch was uploaded. Can I still find that old job? | 23:18 |
| *** gnuoy has joined #openstack-infra | 23:19 | |
| *** jamespage has joined #openstack-infra | 23:19 | |
| *** yamamoto has joined #openstack-infra | 23:20 | |
| openstackgerrit | Merged openstack-infra/puppet-haveged: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183261 | 23:20 |
| clarkb | jlvillal: yes, simplest way is probably to search for it in logstash, but the log urls are deterministic. so for http://logs.openstack.org/95/182895/1/check/gate-puppet-elasticsearch-puppet-lint/c385280/ its last 2 of change number/change number/patchset/pipeline/jobname/job-uuids | 23:20 |
| openstackgerrit | Merged openstack-infra/puppet-kibana: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183267 | 23:21 |
| jlvillal | clarkb: Thanks. I will go look. | 23:21 |
| openstackgerrit | Merged openstack-infra/puppet-logrotate: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183270 | 23:21 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183273 | 23:21 |
| openstackgerrit | Merged openstack-infra/shade: Improve error message on auth_plugin failure https://review.openstack.org/182898 | 23:21 |
| openstackgerrit | Merged openstack-infra/puppet-mailman: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183272 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183265 | 23:22 |
| *** hdd has joined #openstack-infra | 23:22 | |
| openstackgerrit | Merged openstack-infra/puppet-iptables: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183263 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-meetbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183274 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-mysql_backup: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183275 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183276 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183279 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-packagekit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183280 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-openafs: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183277 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-pip: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183281 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-planet: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183282 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-simpleproxy: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183287 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-reviewday: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183286 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-releasestatus: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183285 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-project_config: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183283 | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-redis: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183284 | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config: Group akanda sub-projects https://review.openstack.org/183146 | 23:23 |
| mordred | merge bomb | 23:23 |
| mordred | jhesketh, clarkb: I think we should be able to do this one now: 167610 | 23:25 |
| *** atan8 has joined #openstack-infra | 23:26 | |
| *** bhuvan has quit IRC | 23:26 | |
| mordred | clarkb: also, I went ahead and dropped my -1 on your other one - I still think we should go do _something_ there | 23:26 |
| *** bhuvan has joined #openstack-infra | 23:26 | |
| mordred | but maybe not today | 23:26 |
| *** banix has quit IRC | 23:26 | |
| *** atan8 has quit IRC | 23:26 | |
| *** yamamoto has quit IRC | 23:27 | |
| fungi | yeah, retrying image updates in hpcloud, it's completely hosed in there at the moment | 23:28 |
| fungi | i guess their maintenance is still underway | 23:29 |
| *** enikanorov2 has quit IRC | 23:29 | |
| *** Longgeek has quit IRC | 23:30 | |
| *** Longgeek has joined #openstack-infra | 23:30 | |
| *** Sukhdev has quit IRC | 23:30 | |
| clarkb | mordred: the removal of python-yaml seems odd since nodepool depends on that too | 23:31 |
| *** banix has joined #openstack-infra | 23:31 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:33 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Install lxml from source to accomodate pip -U https://review.openstack.org/182830 | 23:33 |
| *** Krinkle is now known as Krinkle|detached | 23:33 | |
| clarkb | mordred: are we sure that removing that won't make nodepool unhappy? | 23:33 |
| clarkb | mordred: otherwise it looks good, the diskimage_builder module is installed etc | 23:34 |
| openstackgerrit | Merged openstack-infra/puppet-ssh: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183289 | 23:35 |
| mordred | clarkb: nodepool is going to get it from its pip install anyway though, no? | 23:35 |
| mordred | (pyyaml) | 23:35 |
| *** dimtruck is now known as zz_dimtruck | 23:36 | |
| clarkb | mordred: ya, so why would we need to have added the python-yaml package? | 23:36 |
| openstackgerrit | Merged openstack-infra/puppet-ssl_cert_check: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183290 | 23:37 |
| openstackgerrit | Merged openstack-infra/puppet-statusbot: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183291 | 23:37 |
| openstackgerrit | Merged openstack-infra/puppet-snmpd: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183288 | 23:37 |
| * fungi wonders if that package was a hold-over from when we weren't doing pip install -U | 23:38 | |
| *** zhiwei has joined #openstack-infra | 23:39 | |
| mordred | I would vote on that ^^ | 23:39 |
| fungi | for a while there, iirc, we installed distro packages for any python libs we could and just relied on pip to fill in the otherwise unpackaged gaps | 23:39 |
| mordred | clarkb: where did we get on nibalizer's server/template stuff? | 23:41 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183292 | 23:41 |
| openstackgerrit | Merged openstack-infra/puppet-lodgeit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183268 | 23:41 |
| clarkb | mordred: its up for review, I retopic'd it then VENOM and all that fun happened | 23:41 |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183271 | 23:41 |
| openstackgerrit | Merged openstack-infra/puppet-log_processor: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183269 | 23:41 |
| *** bhuvan has quit IRC | 23:42 | |
| mordred | kk | 23:42 |
| *** angela-s has left #openstack-infra | 23:43 | |
| mordred | oh - they both landed | 23:44 |
| clarkb | awesome | 23:44 |
| mordred | so now next step is moving things from server to template? | 23:44 |
| clarkb | ya to flip the manifest they use as their base | 23:45 |
| mordred | because server and template _should_ be equiv now, yeah? | 23:45 |
| clarkb | mordred: I +2'd the dib module one | 23:45 |
| mordred | cool | 23:45 |
| clarkb | mordred: server calls template so ya its just an unnecessary layer | 23:45 |
| mordred | oh - server is just calling template | 23:46 |
| mordred | well, there is one more - we now need to pass the exim flag to template in server | 23:46 |
| *** ashleighfarnham has quit IRC | 23:48 | |
| openstackgerrit | Merged openstack-infra/zuul: Fix git prune order in zuul-cloner https://review.openstack.org/183348 | 23:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Pass exim flag to template rather than exim class https://review.openstack.org/183357 | 23:49 |
| clarkb | greghaynes: one we get new images built (tomorrow morning assuming hpcloud cooperates) you can remove your git clones from the job | 23:49 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183293 | 23:49 |
| *** ajmiller_ has quit IRC | 23:51 | |
| greghaynes | clarkb: ok | 23:51 |
| clarkb | fungi: is hpcloud just not making images at all? | 23:51 |
| nibalizer | mordred: yep so now its 'git grep openstack_server' and find and fix things | 23:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Pass exim flag to template rather than exim class https://review.openstack.org/183357 | 23:52 |
| nibalizer | which is gonna be a fir bit of churn | 23:52 |
| nibalizer | oh that looks interesting | 23:52 |
| clarkb | but we can constrain it which is the nice thing | 23:52 |
| *** rguillebert has quit IRC | 23:52 | |
| openstackgerrit | Merged openstack-infra/puppet-tmpreaper: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183295 | 23:54 |
| openstackgerrit | Merged openstack-infra/puppet-sudoers: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183294 | 23:54 |
| openstackgerrit | Merged openstack-infra/puppet-ulimit: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183296 | 23:54 |
| openstackgerrit | Merged openstack-infra/puppet-unattended_upgrades: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183297 | 23:54 |
| mordred | yup | 23:54 |
| *** _nadya_ has joined #openstack-infra | 23:54 | |
| openstackgerrit | Merged openstack-infra/puppet-unbound: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183298 | 23:56 |
| *** _nadya_ has quit IRC | 23:58 | |
| *** ildikov has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!