| * fungi has not-so-fond memories of those days | 00:00 | |
| clarkb | oh does bash do funny things to locals outside of functions? | 00:00 |
|---|---|---|
| clarkb | help local says yes | 00:00 |
| clarkb | fun | 00:00 |
| fungi | and yes, i _think_ 186449 should solve the issue we saw | 00:00 |
| fungi | basically jobs began failing because the variable pointing into the venv bindir was empty | 00:00 |
| fungi | for post-test things like pip freeze and so on | 00:01 |
| fungi | AJaeger and pabelanger ran that down and saved the day while i was rolling back images | 00:01 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Replace ci.o.o links with docs.o.o/infra https://review.openstack.org/183239 | 00:02 |
| SpamapS | clarkb: "locals outside of functions" .. this doesn't make sense to me? | 00:02 |
| clarkb | SpamapS: see 186449 | 00:02 |
| clarkb | SpamapS: it doesn't treat local as being gobal in that case | 00:02 |
| clarkb | and the help text basically says don't do this | 00:02 |
| greghaynes | clarkb: huh, it should error I thought | 00:03 |
| greghaynes | if its not in a function | 00:03 |
| SpamapS | clarkb: yeah. :) | 00:03 |
| SpamapS | /tmp/test.bash: line 1: local: can only be used in a function | 00:04 |
| greghaynes | oh, that script is not -e | 00:04 |
| SpamapS | Probably getting sourced? | 00:04 |
| *** kutija has quit IRC | 00:04 | |
| anteaya | adam_g: why the underscore in the job name? I think our jobs use hyphens | 00:04 |
| *** sarob has quit IRC | 00:04 | |
| *** wenlock_ has quit IRC | 00:04 | |
| SpamapS | greghaynes: probably sourced from inside a function | 00:04 |
| greghaynes | ah | 00:04 |
| jeblair | adam_g: +2 with suggestions for future enhancements | 00:05 |
| *** shashankhegde has quit IRC | 00:06 | |
| adam_g | anteaya, good call | 00:06 |
| clarkb | jeblair: fwiw we likely need a better story for all these large images we are caching | 00:06 |
| clarkb | jeblair: particularly since the image we upload to rax is 14GB | 00:06 |
| clarkb | still uploading now almost two hours later \o/ | 00:06 |
| adam_g | jeblair, thanks. both your suggestions are immediate next steps once these are functional | 00:06 |
| *** Kennan is now known as Kennan_ | 00:06 | |
| jeblair | clarkb: it takes 2 hours to move 14G? also, that's a lot of G. | 00:07 |
| *** yamada-h has joined #openstack-infra | 00:07 | |
| clarkb | jeblair: well it hasn't finished yet so not sure how long it takes and I only just noticed the size so haven't fully dug in there yet | 00:07 |
| clarkb | but I don't think just cache $image on slave image is going to be viable for much longer | 00:08 |
| greghaynes | is it mostly packages? | 00:08 |
| jeblair | root@nodepool:~# nodepool list | 00:08 |
| jeblair | os_client_config.exceptions.OpenStackConfigException: Named cloud rax requested that was not found. | 00:08 |
| jeblair | :( | 00:08 |
| clarkb | jeblair: you have to sudo -H -u nodepool | 00:08 |
| clarkb | the os-client-config stuff requires it | 00:08 |
| jeblair | ok | 00:09 |
| clarkb | greghaynes: no I think its mostly git and the images we cache | 00:09 |
| clarkb | greghaynes: git is ~4GB last I checked | 00:09 |
| greghaynes | wow | 00:09 |
| greghaynes | clarkb: If you convert the image to v3 qcow2 I wonder how much smaller it gets... | 00:09 |
| openstackgerrit | Merged openstack-infra/project-config: Update more infra project ACLs https://review.openstack.org/186595 | 00:10 |
| jeblair | clarkb: i guess we could stop caching images, or have a policy of only caching heavily used ones under a certain size... | 00:10 |
| sc68cal | anteaya: thanks | 00:10 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Adds initial akanda dsvm functional tests https://review.openstack.org/182391 | 00:10 |
| anteaya | sc68cal: welcome is it back yet? | 00:10 |
| greghaynes | Maybe put images in the per-region mirrors? | 00:10 |
| adam_g | jeblair, anteaya take two ^ | 00:10 |
| clarkb | greghaynes: well the backward compat qcow2 is ~4.9GB | 00:10 |
| greghaynes | clarkb: oh, right, its vhd in rax | 00:11 |
| greghaynes | clarkb: ugh | 00:11 |
| fungi | clarkb: greghaynes: yeah, this is the "must provide raw images" thing | 00:11 |
| *** davideagnello has quit IRC | 00:11 | |
| clarkb | anteaya: fungi for https://review.openstack.org/#/c/186202/2 was that controversial? the change looks fine to me but it wasn't clear to me if we thought there was more discussion needed | 00:11 |
| fungi | there's a related thread running on either the dev or ops ml right now, though specific to ceph | 00:11 |
| sc68cal | anteaya: not sure I'll look, just got back from workout | 00:11 |
| *** yamada-h has quit IRC | 00:12 | |
| clarkb | I think clouds should take a compressed easy to use format (eg qcow2) then convert it to whatever format is apprpriate for their hypervisor/volumes systems | 00:12 |
| greghaynes | the glance people at summit said that image conversion on the server works now | 00:12 |
| *** markvoelker has quit IRC | 00:12 | |
| greghaynes | not sure what all is entailed though | 00:12 |
| anteaya | clarkb: I don't find it controvertial just getting enough support for it to get it approved is my motivation | 00:13 |
| clarkb | greghaynes: I don't think any clouds support it | 00:13 |
| clarkb | anteaya: cool will approve now | 00:13 |
| anteaya | clarkb: my early question was my way of generating support | 00:13 |
| anteaya | thanks | 00:13 |
| anteaya | sc68cal: I don't think it has finished yet, I hope you had a good workout | 00:13 |
| *** rkukura has joined #openstack-infra | 00:13 | |
| clarkb | greghaynes: neat, I wonder if any clouds have deployed it (specifically rax since that is the one that we have all this crazy vhd stuff for) | 00:14 |
| openstackgerrit | Merged openstack-infra/system-config: Set rax image metadata properly https://review.openstack.org/186587 | 00:14 |
| greghaynes | clarkb: yea, would be awesome | 00:14 |
| *** shashankhegde has joined #openstack-infra | 00:14 | |
| fungi | we could pester someone we know at rax to find out | 00:15 |
| anteaya | adam_g: did you read jeblair's comments? I don't see them addressed in your latest patchset | 00:15 |
| jeblair | anteaya: oh they were only for future work | 00:16 |
| adam_g | anteaya, as he noted here, future improvements. getting build artifacts published to tarballs.o.o was my next order of business once they're builds are actually happenin | 00:16 |
| anteaya | jeblair: ah | 00:17 |
| *** chuckC_ has quit IRC | 00:17 | |
| anteaya | adam_g: okey dokey then | 00:17 |
| anteaya | away it goes | 00:17 |
| adam_g | thanks ! | 00:17 |
| *** gyee has quit IRC | 00:18 | |
| anteaya | welcome | 00:18 |
| *** puranamr has quit IRC | 00:20 | |
| *** dimtruck is now known as zz_dimtruck | 00:20 | |
| *** Sukhdev has quit IRC | 00:21 | |
| *** sputnik13 has quit IRC | 00:21 | |
| *** cody-somerville has quit IRC | 00:23 | |
| *** emagana has quit IRC | 00:25 | |
| *** gyee has joined #openstack-infra | 00:25 | |
| *** wenlock has quit IRC | 00:25 | |
| *** SotK has quit IRC | 00:30 | |
| *** SotK has joined #openstack-infra | 00:30 | |
| *** salv-orlando has joined #openstack-infra | 00:31 | |
| *** salv-orlando has quit IRC | 00:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Rename irc-meetings index.html in job logs https://review.openstack.org/186202 | 00:35 |
| *** sdake_ has quit IRC | 00:38 | |
| *** sdake has joined #openstack-infra | 00:38 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack/requirements: Bump docker-py version for new exec call https://review.openstack.org/186610 | 00:39 |
| *** yamahata has joined #openstack-infra | 00:39 | |
| *** tjones1 has joined #openstack-infra | 00:41 | |
| *** ociuhandu has quit IRC | 00:41 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:41 | |
| *** dims_ has quit IRC | 00:42 | |
| *** tjones1 has joined #openstack-infra | 00:44 | |
| *** annegentle has joined #openstack-infra | 00:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add diskimage-builder notification to infra https://review.openstack.org/186614 | 00:46 |
| *** dims_ has joined #openstack-infra | 00:47 | |
| *** mdrnstm has joined #openstack-infra | 00:48 | |
| *** mdrnstm is now known as Guest723 | 00:48 | |
| *** Longgeek has joined #openstack-infra | 00:49 | |
| *** Kennan_ is now known as Kennan | 00:50 | |
| *** tjones1 has quit IRC | 00:51 | |
| *** SumitNaiksatam has quit IRC | 00:52 | |
| *** sdake_ has joined #openstack-infra | 00:52 | |
| *** Guest723 has quit IRC | 00:55 | |
| *** tiswanso has joined #openstack-infra | 00:55 | |
| *** shashankhegde has quit IRC | 00:55 | |
| *** tiswanso has quit IRC | 00:57 | |
| *** sdake has quit IRC | 00:57 | |
| *** tiswanso has joined #openstack-infra | 00:57 | |
| *** koolhead17 has joined #openstack-infra | 00:58 | |
| *** koolhead17 has quit IRC | 00:58 | |
| *** koolhead17 has joined #openstack-infra | 00:58 | |
| *** Kennan has quit IRC | 00:58 | |
| *** Kennan has joined #openstack-infra | 00:59 | |
| *** vahidh has quit IRC | 00:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add Fedora 22 DIB nodes https://review.openstack.org/186619 | 01:00 |
| *** koolhead17 has quit IRC | 01:00 | |
| *** otter768 has joined #openstack-infra | 01:01 | |
| *** shashankhegde has joined #openstack-infra | 01:02 | |
| *** annegentle has quit IRC | 01:03 | |
| *** gokrokve has joined #openstack-infra | 01:03 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-log_processor: Remove dependency on logstash::indexer class https://review.openstack.org/186621 | 01:03 |
| *** otter768 has quit IRC | 01:06 | |
| *** yamamoto has quit IRC | 01:06 | |
| *** shashankhegde has quit IRC | 01:06 | |
| *** ayoung has joined #openstack-infra | 01:08 | |
| *** alexsyip has quit IRC | 01:08 | |
| mriedem | is this a known issue? no pip in the py26 jobs on stable? http://logs.openstack.org/28/186428/1/check/gate-oslo-incubator-python26/6bed457/console.html#_2015-05-28_15_26_30_796 | 01:08 |
| mriedem | looks like multiple jobs http://goo.gl/q3Eg0l | 01:10 |
| *** tiswanso has quit IRC | 01:12 | |
| *** ddieterly has joined #openstack-infra | 01:12 | |
| *** markvoelker has joined #openstack-infra | 01:13 | |
| lifeless | thats new to me | 01:14 |
| *** koolhead17 has joined #openstack-infra | 01:16 | |
| dstufft | The /pip thing looks suspect possibly | 01:17 |
| *** cody-somerville has joined #openstack-infra | 01:17 | |
| *** sdake_ has quit IRC | 01:17 | |
| dstufft | presumably that should be "pip" not "/pip" | 01:17 |
| dstufft | or /something/pip | 01:17 |
| dstufft | and the envvar for a base env is empty maybe | 01:18 |
| *** markvoelker has quit IRC | 01:18 | |
| *** sdake has joined #openstack-infra | 01:18 | |
| *** gokrokve has quit IRC | 01:20 | |
| *** gokrokve has joined #openstack-infra | 01:20 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:22 | |
| *** GheRivero has joined #openstack-infra | 01:24 | |
| *** gokrokve has quit IRC | 01:25 | |
| *** ivar-laz_ has quit IRC | 01:25 | |
| *** doug-fish has left #openstack-infra | 01:26 | |
| *** ivar-lazzaro has quit IRC | 01:26 | |
| *** alexpilotti has joined #openstack-infra | 01:28 | |
| mriedem | looks intermittent | 01:29 |
| *** zhiwei has joined #openstack-infra | 01:31 | |
| *** emagana has joined #openstack-infra | 01:32 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-log_processor: Break up logstash::indexer and log_processor::worker https://review.openstack.org/186621 | 01:35 |
| *** dims_ has quit IRC | 01:35 | |
| *** yamahata has quit IRC | 01:36 | |
| *** alexpilotti has quit IRC | 01:38 | |
| *** dims_ has joined #openstack-infra | 01:39 | |
| clarkb | mriedem that was due to an image build change old images were removed | 01:39 |
| *** Kiall has quit IRC | 01:41 | |
| *** Kiall_ has joined #openstack-infra | 01:41 | |
| *** yamamoto has joined #openstack-infra | 01:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds initial akanda dsvm functional tests https://review.openstack.org/182391 | 01:44 |
| *** dims_ has quit IRC | 01:45 | |
| *** amotoki has joined #openstack-infra | 01:46 | |
| mriedem | clarkb: ah ok, good to know. i rechecked and looks like things are passing | 01:46 |
| *** amotoki_ has joined #openstack-infra | 01:47 | |
| *** aduarte has quit IRC | 01:48 | |
| anteaya | our irc-meeting log fix doesn't seem to work: https://review.openstack.org/#/c/186184/ | 01:49 |
| *** bradjones has quit IRC | 01:52 | |
| *** amotoki_ has quit IRC | 01:52 | |
| *** patrickeast has quit IRC | 01:54 | |
| *** bradjones has joined #openstack-infra | 01:56 | |
| *** emagana has quit IRC | 01:57 | |
| *** emagana has joined #openstack-infra | 01:58 | |
| *** tiswanso has joined #openstack-infra | 02:00 | |
| *** unicell has quit IRC | 02:07 | |
| *** yamada-h has joined #openstack-infra | 02:09 | |
| *** ddieterly has quit IRC | 02:13 | |
| *** yamada-h has quit IRC | 02:14 | |
| *** shashankhegde has joined #openstack-infra | 02:15 | |
| *** Longgeek has quit IRC | 02:16 | |
| Kennan | hi, anyone know this issue | 02:16 |
| Kennan | 2015-05-29 01:41:26.344 | ('No versions of flake8 could have its requirements met.', ['flake8', 'hacking', 'flake8'], 184) | 02:16 |
| lifeless | it should have been sorted last week | 02:19 |
| lifeless | but perhaps you could link to the line in the log that would give more context | 02:19 |
| *** wenlock has joined #openstack-infra | 02:19 | |
| *** mriedem has quit IRC | 02:20 | |
| *** rwsu has quit IRC | 02:25 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add Python 3.4 as a non-voting check to Bandit https://review.openstack.org/186629 | 02:32 |
| Kennan | lifeless: http://logs.openstack.org/10/186610/1/check/check-requirements-integration-dsvm-resolver/f90cf62/console.html | 02:33 |
| lifeless | Kennan: thats the resolver job, its nonvoting. | 02:34 |
| lifeless | Kennan: https://review.openstack.org/#/c/183930/ will fix that particular issue, once I get the attention of g-r cores. | 02:35 |
| lifeless | mordred: clarkb: ^ | 02:35 |
| Kennan | lifeless: also noticed your review | 02:37 |
| Kennan | 2015-05-19 00:02:11.356 | No hosts matched | 02:37 |
| Kennan | 2015-05-19 00:02:11.367 | + sudo mv /home/jenkins/workspace/check-requirements-integration-dsvm-resolver/devstack-gate-cleanup-host.txt /opt/stack/logs/ | 02:37 |
| Kennan | 2015-05-19 00:02:11.374 | + exit 1 | 02:37 |
| Kennan | 2015-05-19 00:02:11.449 | Build step 'Execute shell' marked build as failure | 02:37 |
| Kennan | it is annonying why resovled always failed | 02:37 |
| lifeless | why? | 02:38 |
| *** sdake_ has joined #openstack-infra | 02:40 | |
| *** sdake has quit IRC | 02:43 | |
| *** sdake has joined #openstack-infra | 02:43 | |
| *** sdake_ has quit IRC | 02:47 | |
| *** jyuso has quit IRC | 02:48 | |
| *** yamada-h has joined #openstack-infra | 02:49 | |
| Kennan | lifeless: https://review.openstack.org/#/c/183930/ | 02:50 |
| Kennan | the resolver failed | 02:50 |
| lifeless | yes, and ? | 02:50 |
| lifeless | why is that annoying to you | 02:50 |
| lifeless | Kennan: ^ | 02:51 |
| Kennan | it means I thought my review or the jenkins system has issue | 02:52 |
| Kennan | need to fix | 02:52 |
| Kennan | :) | 02:52 |
| lifeless | no | 02:52 |
| lifeless | its a non-voting job | 02:52 |
| lifeless | see the 'non-voting' beside it | 02:52 |
| Kennan | yes lifeless: it is non-voting if it is not useful, why need it :) | 02:52 |
| lifeless | it is useful | 02:52 |
| lifeless | I still don't understand why its failing is annoying to you | 02:53 |
| lifeless | would you like to help develop the resolver for pip ? | 02:53 |
| *** samueldmq has quit IRC | 02:53 | |
| *** ddieterly has joined #openstack-infra | 02:53 | |
| lifeless | you're more than welcome to join in, I'd be happy to have help | 02:53 |
| lifeless | though its tricky to scale out, as its basically one work item | 02:53 |
| lifeless | I'm sure we can work something out | 02:53 |
| Kennan | lifeless: ok, I will check if possible to fix the failure in jenkins | 02:53 |
| Kennan | I would be happy to find all passed (if no code issue) | 02:54 |
| lifeless | Kennan: this isn't a code issue per se | 02:54 |
| Kennan | ok | 02:55 |
| lifeless | Kennan: our requirements aren't co-installable, and we need to fix that. | 02:55 |
| lifeless | the job will fail while they are, or if the resolver finds the problem too hard | 02:55 |
| lifeless | (its Non-polynomial complete0 | 02:55 |
| lifeless | Kennan: but *in general* you should ignore non-voting jobs unless you're working on them | 02:56 |
| lifeless | because they are non-voting because they are known to be not yet ready to make voting | 02:56 |
| openstackgerrit | Qin Zhao proposed openstack/requirements: Update sqlalchemy-migrate to 0.9.6 https://review.openstack.org/181821 | 02:57 |
| *** _ody has quit IRC | 03:00 | |
| *** mdrnstm has joined #openstack-infra | 03:01 | |
| *** otter768 has joined #openstack-infra | 03:01 | |
| *** markvoelker has joined #openstack-infra | 03:02 | |
| *** gyee is now known as operator99 | 03:06 | |
| *** markvoelker has quit IRC | 03:06 | |
| *** wenlock has quit IRC | 03:06 | |
| *** otter768 has quit IRC | 03:07 | |
| *** yamada-h has quit IRC | 03:07 | |
| *** fawadkhaliq has joined #openstack-infra | 03:07 | |
| *** yamada-h has joined #openstack-infra | 03:09 | |
| *** davideagnello has joined #openstack-infra | 03:09 | |
| *** yamada-h has joined #openstack-infra | 03:10 | |
| *** sputnik13 has joined #openstack-infra | 03:10 | |
| *** davideagnello has quit IRC | 03:14 | |
| *** jerryz has quit IRC | 03:22 | |
| lifeless | boom | 03:22 |
| nikeshm | hi | 03:27 |
| nikeshm | i want to add http://docs.openstack.org/infra/system-config/third_party.html#faq-frequently-asked-questions in apache web server | 03:27 |
| nikeshm | i am adding in apache2.conf file and restarted but getting this error http://paste.openstack.org/show/243757/ | 03:29 |
| mdrnstm | lifeless: oh hai | 03:29 |
| *** btully has quit IRC | 03:29 | |
| lifeless | mdrnstm: hi | 03:29 |
| lifeless | mdrnstm: btw https://review.openstack.org/186635 | 03:30 |
| mdrnstm | lifeless: ooooooooooh | 03:30 |
| nibalizer | nikeshm: looks like you need to install mod_rewrite | 03:30 |
| nibalizer | an apache plugin | 03:31 |
| openstackgerrit | Ian Cordasco proposed openstack-dev/hacking: WIP: Add check for use of unicode type instead of six.text_type https://review.openstack.org/186637 | 03:31 |
| mdrnstm | lifeless: nice | 03:31 |
| *** bhunter71 has quit IRC | 03:31 | |
| *** tonytan4ever has quit IRC | 03:33 | |
| mdrnstm | lifeless: OMG I see erroneous whitespace /bikeshed | 03:33 |
| mdrnstm | lifeless: in all seriousness, at a glance, looks good! | 03:33 |
| * mdrnstm likes | 03:33 | |
| mordred | lifeless: wow. we're still doing that one? | 03:34 |
| * mdrnstm thinks it is silly we *need* this. but clearly stating how the pins/requirements etc work will help | 03:35 | |
| mdrnstm | and not breaking all the time is the part i like the best | 03:35 |
| *** harlowja_at_home has joined #openstack-infra | 03:35 | |
| *** amotoki_ has joined #openstack-infra | 03:36 | |
| *** shashankhegde has quit IRC | 03:37 | |
| lifeless | mordred: still doing what one? | 03:37 |
| *** puranamr has joined #openstack-infra | 03:37 | |
| nikeshm | nibalizer : thanks sudo a2enmod rewrite after this working | 03:37 |
| nikeshm | :) | 03:37 |
| lifeless | mordred: oh right the g-r thing, yes. :) | 03:37 |
| nibalizer | woot | 03:40 |
| lifeless | mdrnstm: I'm sure there are bits I missed | 03:40 |
| lifeless | mdrnstm: so were you saying hi to me with an agenda? Or just saw my boom? | 03:41 |
| mdrnstm | lifeless: just saying hi | 03:41 |
| mdrnstm | lifeless: no agenda (yet) | 03:41 |
| lifeless | :) | 03:41 |
| *** amotoki_ has quit IRC | 03:41 | |
| mdrnstm | lifeless: trying to figure out how to kill oslo.config from keystoneauth | 03:41 |
| lifeless | mdrnstm: oh? | 03:42 |
| mdrnstm | lifeless: but that isn't really something i need your brain for...it's just a "how do we do things w/o config" and fix it | 03:42 |
| mdrnstm | lifeless: yeah we're close to being free of all oslo libs in ksa | 03:42 |
| mdrnstm | http://git.openstack.org/cgit/openstack/keystoneauth/tree/requirements.txt | 03:42 |
| lifeless | mdrnstm: I didn't realise that was a goal | 03:42 |
| *** yamada-h has joined #openstack-infra | 03:42 | |
| mdrnstm | lifeless: keystoneauth is planned to have as minimal a requirement set as possible so we can use it all-the-places (SDK, server for service-to-service, and in clients) | 03:43 |
| clarkb | just use config parser and argparse? | 03:43 |
| mdrnstm | clarkb: not sure we even need that. | 03:43 |
| clarkb | ah even better then | 03:43 |
| mdrnstm | clarkb: yeah :) | 03:43 |
| lifeless | mdrnstm: I don't follow how those two things are connected? | 03:43 |
| mdrnstm | lifeless: oslo libs introduce an odd dependency graph | 03:44 |
| lifeless | mdrnstm: the overly tight install_requires would have caused issues, but thats because thats expected... | 03:44 |
| lifeless | mdrnstm: and will be fixed this cycle. | 03:44 |
| *** yamada-h has joined #openstack-infra | 03:44 | |
| mdrnstm | we want to avoid conflicts so updating SDK for example on the same system where keystoneauth2 (future second major release) wont cause issues with a service that is older | 03:44 |
| mdrnstm | or a client | 03:44 |
| *** salv-orlando has joined #openstack-infra | 03:45 | |
| lifeless | mdrnstm: right - so that was the original premise | 03:45 |
| mdrnstm | it's really about ensuring we have a minimal set of dependencies and keep it as close to standard libs as possible | 03:45 |
| lifeless | mdrnstm: I'm still not 100% clear on why we stopped testing that that setup could work | 03:45 |
| lifeless | it was one of the things I meant to drill into last week | 03:45 |
| lifeless | but ETIME | 03:45 |
| mdrnstm | lifeless: ETIME is a real issue at ${summit} | 03:46 |
| lifeless | clarkb: oh btw - https://review.openstack.org/186635 - is a thing | 03:46 |
| *** viglesias has quit IRC | 03:46 | |
| mdrnstm | lifeless: anywya i want keystoneauth to be really really minimal sets of deps - it also keeps us from growing magical load time(s) that are unexpected | 03:46 |
| mdrnstm | lifeless: even if the dep graph issues aren't an issue | 03:46 |
| clarkb | lifeless yup abojt to leave a comment | 03:46 |
| lifeless | clarkb: during the discussion for that your name came up a couple of times.. one in particular was the question of how to tie in upper-constraints.txt to tox | 03:47 |
| mdrnstm | why does KSA need to know how to read oslo.config things becomes a real question | 03:47 |
| *** salv-orlando has quit IRC | 03:47 | |
| *** viglesias has joined #openstack-infra | 03:47 | |
| lifeless | mdrnstm: shrug - sure, not really arguing, just trying to understand. | 03:47 |
| lifeless | mdrnstm: I guess my fear is that we end up not using shared libs because there's a fundamental thing we haven't solved. | 03:47 |
| lifeless | mdrnstm: e.g. 'how to test old releases with new ksa' [or whatever] | 03:48 |
| clarkb | lifeless comment on ps1 mostly minor | 03:48 |
| clarkb | ya the how to tox question is hard particularly aince its not in an integration test env | 03:48 |
| mdrnstm | lifeless: sure. we should be testing that too. but we also shouldn't depend on oslo.utils for 1 datetime parsing function, or oslo.serialization if we don't really need magic that json stdlib can't do. | 03:49 |
| clarkb | we could do the naive delayed updates like for g-r | 03:49 |
| mdrnstm | (esp. because msgpack) | 03:49 |
| mdrnstm | things like that | 03:49 |
| clarkb | then install that in tox instead of requirements.txt | 03:49 |
| clarkb | and call that "close enough" | 03:49 |
| clarkb | it should always work just not always be latest | 03:49 |
| lifeless | clarkb: so IIRC the idea we had was a) an env variable specifying the path, b) zuul-cloner to get the file on disk | 03:50 |
| lifeless | clarkb: then folks at home could opt in by cloning requirements and setting the variable, and we can look at making it super easy to opt in at home as a future iteration | 03:50 |
| clarkb | hrm that becomes complicated for local tox. makes zuul-cloner a dep everywhere and todat that requires all of zuul. fixable though | 03:51 |
| lifeless | but we felt that breaking folk by making existing setups not work would be bad | 03:51 |
| clarkb | oh I see | 03:51 |
| *** puranamr has quit IRC | 03:51 | |
| mdrnstm | lifeless: ++ | 03:51 |
| lifeless | clarkb: (with the variable not set meaning no constraints applied) | 03:51 |
| clarkb | we could also do the syncing to have mostly up to date constraints | 03:52 |
| clarkb | but still opt in | 03:52 |
| lifeless | clarkb: this may imply that we need something to interpret that variable within the venv - a wrapper to pip or something | 03:52 |
| lifeless | clarkb: but we're likely going to need that for a while anyway, depending on exactly how the constraint implementation is solved. | 03:52 |
| lifeless | clarkb: I'm thinking to put that in os-testr's repo | 03:52 |
| clarkb | I dont like that because I dont want to use os-testr :) | 03:53 |
| clarkb | (not a fan of its opinions) | 03:53 |
| lifeless | clarkb: I'd very much like to avoid having copies of upper-constraints.txt in each repo | 03:53 |
| *** achanda has joined #openstack-infra | 03:53 | |
| mdrnstm | lifeless: i support not needing to copy things like that if we can avoid it. | 03:53 |
| *** tiswanso has quit IRC | 03:53 | |
| lifeless | clarkb: in particular, we have a hard requirement that the gate not be skewed at any point to avoid needle threading opportunities | 03:54 |
| mdrnstm | lifeless: i have that spec open, will comment on it tonight once i'm done with preso things. | 03:54 |
| lifeless | clarkb: and having local copies could easily lead to bugs where a stale copy is consulted | 03:54 |
| lifeless | mdrnstm: cool | 03:54 |
| clarkb | lifeless no because its not integration | 03:54 |
| clarkb | so no needle threading | 03:54 |
| clarkb | with devstack yes, but not running pep8 or unittests | 03:55 |
| lifeless | if its in the repos, then devstack will have multiple copies of it available, and a bug could pick one of those up | 03:55 |
| lifeless | is what I'm sayin | 03:55 |
| lifeless | even in the dvsm runs | 03:55 |
| lifeless | clarkb: the freeze file size doesn't change much | 03:56 |
| mordred | I support all the things | 03:56 |
| lifeless | clarkb: it turns out we have very few transitive only deps today | 03:56 |
| *** yamada-h has quit IRC | 03:56 | |
| lifeless | because someone somewhere uses $thing for nearly all $things. | 03:56 |
| clarkb | huh thats interesting | 03:57 |
| lifeless | 268 ../requirements/global-requirements.txt | 03:57 |
| lifeless | IIRC pip found 275 | 03:57 |
| lifeless | something like that | 03:57 |
| lifeless | I mean - we may have transitive-onlies in g-r because we've had to cap them or something | 03:57 |
| clarkb | ya | 03:58 |
| lifeless | but unwinding that is a separate discussion | 03:58 |
| lifeless | [one that should be easier once this is in place] | 03:59 |
| *** viglesias has quit IRC | 03:59 | |
| clarkb | also I think pbr is what should learn about constraints and overlay into requirements at install time? thats my initial how I would do it at least | 03:59 |
| *** viglesias has joined #openstack-infra | 04:00 | |
| *** puranamr has joined #openstack-infra | 04:00 | |
| clarkb | so PBR_DEP_CONSTRAINTS=/some/file and it intersects onto the install requires | 04:00 |
| lifeless | nooo | 04:00 |
| clarkb | oh? | 04:00 |
| lifeless | we just got pbr out of the business of doing installs | 04:00 |
| clarkb | its still not doing an install | 04:01 |
| clarkb | all it does is update the list given to setuptools/pip | 04:01 |
| lifeless | and there's no way to indicate 'constraint' within the setuptools metadata that pip interprets | 04:01 |
| lifeless | only 'requires' | 04:01 |
| clarkb | right you change the requires | 04:01 |
| lifeless | the intersect has to be transitive | 04:01 |
| lifeless | which requires an oracle of per version dependency data | 04:01 |
| lifeless | since we have pbr installed | 04:02 |
| lifeless | we could add a thing to help with the glue etc to pbr's CLI | 04:02 |
| clarkb | gah ok | 04:02 |
| *** fijal_ has joined #openstack-infra | 04:02 | |
| lifeless | but it would be a new thing, not part of pbr's setup.py glue | 04:02 |
| lifeless | possibly a better home than os-testr as you say | 04:02 |
| *** markvoelker has joined #openstack-infra | 04:02 | |
| clarkb | this seems like something pip/setuptools need to support then | 04:03 |
| lifeless | yes | 04:03 |
| lifeless | well | 04:03 |
| clarkb | an install X but consult Y for more info | 04:03 |
| lifeless | no, kindof. | 04:03 |
| lifeless | I likely will implement it by teaching pip about constraints | 04:03 |
| lifeless | I think thats cleanest and fairly low hanging | 04:03 |
| clarkb | ok that makes sense | 04:03 |
| mordred | lifeless: speaking of pbr ... | 04:03 |
| lifeless | but it /can/ be done exterior-wise, its just traumatic :) | 04:03 |
| clarkb | ya it gets hacky fast | 04:04 |
| lifeless | mordred: yes, we need a release on tuesday IIRC. | 04:04 |
| mordred | cool | 04:04 |
| lifeless | mordred: but thats not what you were going to say? | 04:04 |
| lifeless | I haven't fixed the regression setup.py included install_requires yet. Let me make sure there is a bug. | 04:04 |
| mordred | lifeless: https://review.openstack.org/#/c/184899/ <-- I put that together in response to a request that pbr be better at installing manpages. I pointed out that it can quite happily _install_ manpages if they exist - but in most of our projects with manpages they get built by sphinx | 04:04 |
| mordred | so "fix" being - if you run sphinx, pbr will notice the manpages it builds and deal with them. otherwise, it will happily continue to not | 04:05 |
| *** ddieterly has quit IRC | 04:05 | |
| *** achanda has quit IRC | 04:05 | |
| lifeless | speaking of | 04:05 |
| lifeless | mordred: https://review.openstack.org/#/c/178879/ | 04:05 |
| mordred | "python setup.py build_sphinx sdist" should then do what the fine folk are requesting - without injecting sphinx into normal install things | 04:06 |
| *** fijal has quit IRC | 04:06 | |
| lifeless | mordred: I like that approach | 04:06 |
| mordred | awesome | 04:06 |
| *** achanda has joined #openstack-infra | 04:06 | |
| mordred | lifeless: I did not add tests yet - as I wanted to make sure people liked the approach | 04:06 |
| lifeless | mordred: OTOH | 04:06 |
| mdrnstm | mordred: ++ | 04:06 |
| lifeless | mordred: they might be stale | 04:06 |
| mordred | lifeless: I'm willing to live with that for now | 04:07 |
| lifeless | mordred: I'm tempted to say that we should have a way to say call build_sphinx on sdist. | 04:07 |
| lifeless | mordred: (which of course propogates the sphinx deps to setup_requires :)) | 04:07 |
| mordred | lifeless: _maybe_ ... perhaps that should be a v2 | 04:07 |
| lifeless | mordred: yes, living with is fine for now | 04:07 |
| *** _ody has joined #openstack-infra | 04:07 | |
| lifeless | mordred: but lets document the caveat | 04:07 |
| *** markvoelker has quit IRC | 04:07 | |
| lifeless | mordred: REALLY CLEARLY | 04:07 |
| mordred | lifeless: we could have the sdist try to import sphinx, and if so, trigger build_sphinx | 04:08 |
| lifeless | mordred: v1.5. | 04:08 |
| mordred | lifeless: ++ | 04:08 |
| mdrnstm | oo i just notices irssi does some of the rst/md-isms | 04:08 |
| lifeless | I worry that that might just barf in unexpected ways too | 04:08 |
| mordred | aroo? | 04:08 |
| mordred | mdrnstm: oh - it underlined _really_ ? | 04:08 |
| lifeless | mordred: _foo_ | 04:08 |
| mordred | yah | 04:08 |
| mdrnstm | mordred: yeah | 04:08 |
| mordred | google does that too | 04:08 |
| mordred | clarkb: why doesn't weechat do that? | 04:08 |
| StevenK | Yeah, irssi does that | 04:08 |
| mdrnstm | and *bold* | 04:08 |
| lifeless | there's this too foo | 04:08 |
| mordred | clarkb: I don't like it when irrsi has a feature that weechat doesn't - ours is supposed to be cooler | 04:09 |
| StevenK | Perhaps it's configurable | 04:09 |
| lifeless | ctrl-shift-- sayithere ctrl-shift-- | 04:09 |
| lifeless | and you get the underline without the bracketing _'s. | 04:09 |
| mdrnstm | mordred: i forgot weechat was there which is why i was using irssi | 04:09 |
| *** btully has joined #openstack-infra | 04:09 | |
| mordred | weechat smartfilters == goodness | 04:09 |
| * mdrnstm has to be careful, might use 'mdrnstm' as a normal irc nick if i don't reconnect to ZNC soon :P | 04:10 | |
| mdrnstm | but no more mor<tab> fail | 04:10 |
| mordred | mdrnstm: yah. I mean - not that you should avoid mor just because of me | 04:10 |
| *** yamada-h has joined #openstack-infra | 04:11 | |
| mdrnstm | mordred: it's more that digging up the password for znc was more painful than logging in w/o it right now | 04:11 |
| *** achanda has quit IRC | 04:11 | |
| mordred | mdrnstm: but I do like your new 133tr nick | 04:11 |
| mordred | hahaha | 04:11 |
| clarkb | mordred if irssi gets smart filters I might use it again | 04:12 |
| lifeless | oh gosh | 04:12 |
| lifeless | https://bugs.launchpad.net/pbr/+bug/1452930 | 04:12 |
| openstack | Launchpad bug 1452930 in PBR "murano install pbr found then not found isolated private network" [Undecided,Incomplete] | 04:12 |
| clarkb | but must have smart filters | 04:12 |
| *** ajmiller has quit IRC | 04:12 | |
| lifeless | tl;dr - user was editing their setuptools code to tell it where their index was. | 04:12 |
| lifeless | I don't even. | 04:12 |
| mordred | don't do that | 04:12 |
| clarkb | also once I read irssis code, never again | 04:12 |
| lifeless | they also need to learn to paragraph. | 04:12 |
| StevenK | camunoz: I'm having an odd problem -- if I send an Authorization header for GET, it works fine and I get 200 (or 404, depending). If I PUT with the same Authorization header, I get a 401 telling me my API key is invalid. | 04:12 |
| lifeless | (sorry if said user is here... not intending to be mean) | 04:13 |
| lifeless | ok, its dinner cooking time stuff etc | 04:13 |
| lifeless | monday is a pub holiday | 04:13 |
| lifeless | tuesday I'm going to be getting cracking on the requirements work. So bewarned. | 04:14 |
| *** btully has quit IRC | 04:14 | |
| clarkb | I'm out no later than thursday. I wish you much progress | 04:14 |
| *** amotoki has quit IRC | 04:15 | |
| lifeless | clarkb: danke | 04:15 |
| lifeless | clarkb: how long are you out for ? | 04:15 |
| clarkb | not sure yet, I have no idea what to expwct and will probably find myself hacking here and there to not be bored. then again could be completwly wrong | 04:16 |
| lifeless | clarkb: you're taking sabbatical? or parental leave? | 04:16 |
| lifeless | or <other> ? | 04:16 |
| clarkb | ya twins arrive no later than thursday | 04:16 |
| mdrnstm | clarkb: wooooo | 04:17 |
| lifeless | clarkb: oh, I thought they were here already. | 04:17 |
| lifeless | clarkb: boy, you're in for a shock :) | 04:17 |
| clarkb | indeed | 04:17 |
| clarkb | no missing summit was related to complications with pregnancy. so all ties together | 04:18 |
| lifeless | clarkb: best of luck, I predict them hanging facedown one on each arm w/in the week | 04:18 |
| *** mdrnstm has quit IRC | 04:18 | |
| lifeless | [serious - thats a good position for dealing with mild stomach gas, but I'm sure your local support networks will show you :)] | 04:18 |
| camunoz | hi StevenK , can you give me the exact request you are sending? | 04:20 |
| *** ayoung has quit IRC | 04:21 | |
| mordred | lifeless: I responded to your respond on the pbr bug. hopefully between us the human will stop doing evil to themself | 04:21 |
| lifeless | \o/ | 04:21 |
| lifeless | mordred: thanks | 04:21 |
| StevenK | camunoz: http://paste.openstack.org/show/243853/ | 04:22 |
| * camunoz looking | 04:23 | |
| StevenK | camunoz: I'm disappearing to get a (ugh, horribly late) lunch, but I'm very curious if I'm doing something wrong, or the server is. | 04:24 |
| camunoz | StevenK, It's probably just that you don't have permissions to modify the project itself (that's the request that's getting 401), not so much that the api key is wrong.... but I'll make sure and let you know | 04:25 |
| StevenK | camunoz: Said project doesn't exist. I think. | 04:25 |
| camunoz | mmm.... but you are getting something back from the GET | 04:26 |
| StevenK | camunoz: I query for the project itself before trying the PUT | 04:26 |
| StevenK | I will turn full HTTP tracing on after lunch | 04:27 |
| camunoz | ok, so you are trying to create a new project | 04:27 |
| *** dannywilson has quit IRC | 04:28 | |
| *** harlowja_at_home has quit IRC | 04:28 | |
| *** maishsk_afk has quit IRC | 04:29 | |
| *** Sukhdev has joined #openstack-infra | 04:29 | |
| *** sdake_ has joined #openstack-infra | 04:29 | |
| *** sks has joined #openstack-infra | 04:29 | |
| *** fedexo has joined #openstack-infra | 04:31 | |
| *** sputnik13 has quit IRC | 04:31 | |
| *** sdake has quit IRC | 04:33 | |
| *** viglesias has quit IRC | 04:34 | |
| *** Sukhdev has quit IRC | 04:34 | |
| *** viglesias has joined #openstack-infra | 04:35 | |
| *** Longgeek has joined #openstack-infra | 04:36 | |
| *** amotoki has joined #openstack-infra | 04:37 | |
| *** koolhead17 has quit IRC | 04:38 | |
| *** armax has quit IRC | 04:44 | |
| *** Sukhdev has joined #openstack-infra | 04:44 | |
| *** Longgeek has quit IRC | 04:45 | |
| *** Longgeek has joined #openstack-infra | 04:45 | |
| *** jerryz has joined #openstack-infra | 04:45 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add non-voting pypy jobs to taskflow https://review.openstack.org/182782 | 04:48 |
| *** otter768 has joined #openstack-infra | 05:02 | |
| *** achanda has joined #openstack-infra | 05:05 | |
| *** ddieterly has joined #openstack-infra | 05:06 | |
| *** otter768 has quit IRC | 05:07 | |
| *** puranamr has quit IRC | 05:08 | |
| *** ddieterly has quit IRC | 05:10 | |
| *** rossella_s has quit IRC | 05:12 | |
| camunoz | StevenK, please let me know when you're back | 05:13 |
| StevenK | camunoz: Hai | 05:14 |
| camunoz | h Steven, I signed up for an openstack id account, but it seems the Zanata dev instance is authenticating against openstackid-dev | 05:15 |
| *** koolhead17 has joined #openstack-infra | 05:15 | |
| camunoz | there seems no way to get an account for openstackid-dev that I can see | 05:15 |
| StevenK | camunoz: I'm not sure about openstackid-dev TBH | 05:15 |
| StevenK | It's probably using an old database dump | 05:15 |
| camunoz | ahh... makes it kind of difficult to sign in :P | 05:16 |
| *** unicell has joined #openstack-infra | 05:18 | |
| kevinbenton | hello pythonistas, is there a way to break a nested with statement into multiple lines without the dreaded "\" | 05:19 |
| kevinbenton | an advantage of "with contextlib.nested(" was that the open parens let you go multi-line | 05:19 |
| kevinbenton | with the new syntax of "with cm1() as 1, cm2() as b:", i haven't found a way to break it | 05:20 |
| *** Sukhdev has quit IRC | 05:20 | |
| *** Sree_ has joined #openstack-infra | 05:23 | |
| *** Sree_ is now known as Guest60697 | 05:23 | |
| clarkb | use two withs and indent an extra level? | 05:23 |
| *** cody-somerville has quit IRC | 05:23 | |
| clarkb | with (stuff, other stuff) may work too? | 05:23 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting Python 3.4 check for Nova https://review.openstack.org/185363 | 05:24 |
| *** amotoki_ has joined #openstack-infra | 05:25 | |
| clarkb | nope that doesnt seem to work | 05:25 |
| *** yamada-h has quit IRC | 05:29 | |
| *** amotoki_ has quit IRC | 05:30 | |
| *** btully has joined #openstack-infra | 05:31 | |
| *** AzherKhan has joined #openstack-infra | 05:34 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:35 | |
| *** _nadya_ has joined #openstack-infra | 05:36 | |
| *** shashankhegde has joined #openstack-infra | 05:42 | |
| *** sdake has joined #openstack-infra | 05:46 | |
| *** sdake_ has quit IRC | 05:50 | |
| *** markvoelker has joined #openstack-infra | 05:52 | |
| *** mrunge has joined #openstack-infra | 05:53 | |
| *** shashankhegde has quit IRC | 05:54 | |
| *** shashankhegde has joined #openstack-infra | 05:55 | |
| *** shashankhegde has quit IRC | 05:55 | |
| *** hdd has quit IRC | 05:56 | |
| *** markvoelker has quit IRC | 05:56 | |
| *** _nadya_ has quit IRC | 05:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/186650 | 06:02 |
| *** fedexo has quit IRC | 06:03 | |
| *** claudiub has joined #openstack-infra | 06:07 | |
| *** ddieterly has joined #openstack-infra | 06:07 | |
| AJaeger | mordred: your change https://review.openstack.org/#/c/185134/ has merged but the project puppet-phabricator was not created AFAIU | 06:08 |
| StevenK | AJaeger: Did you see the discussion with me and camunoz above? | 06:08 |
| StevenK | AJaeger: Can we get him onto openstackid-dev somehow? | 06:08 |
| kevinbenton | clarkb: yeah, multiple with statements seems to be the only way to go | 06:08 |
| AJaeger | StevenK: not yet.. - and I hvae no idea about openstackid ;( | 06:08 |
| *** cody-somerville has joined #openstack-infra | 06:08 | |
| StevenK | AJaeger: Yeah, that's my problem too. Dratted closed box SSO systems. :-) | 06:09 |
| kevinbenton | clarkb: we have some gnarly mock setups that required like 5 with statements | 06:09 |
| kevinbenton | clarkb: but i just remembered there was something else to deal with that... | 06:09 |
| *** Longgeek has quit IRC | 06:11 | |
| kevinbenton | clarkb: ah i think it was ExitStack() in contextlib | 06:12 |
| *** ddieterly has quit IRC | 06:12 | |
| kevinbenton | clarkb: not any native syntax | 06:12 |
| *** zhiwei has left #openstack-infra | 06:13 | |
| *** jtomasek has joined #openstack-infra | 06:13 | |
| *** _nadya_ has joined #openstack-infra | 06:13 | |
| *** _nadya_ has quit IRC | 06:16 | |
| *** Longgeek has joined #openstack-infra | 06:16 | |
| StevenK | camunoz: http://paste.openstack.org/show/244076/ | 06:19 |
| StevenK | camunoz: The "Project nova is not registered." is from the script itself, indicating it will attempt to register nova. | 06:20 |
| camunoz | StevenK, I think you are missing the request headers for PUT: | 06:20 |
| camunoz | X-Auth-User for username | 06:21 |
| camunoz | and X-Auth-Token for the API key | 06:21 |
| *** akscram has quit IRC | 06:22 | |
| *** akscram has joined #openstack-infra | 06:22 | |
| StevenK | camunoz: Huh. | 06:23 |
| StevenK | camunoz: Why is it Basic auth for GET and not for PUT? | 06:23 |
| camunoz | GET is available for all (the basic auth there is not doing anything really), but PUT needs to be authenticated | 06:24 |
| *** hdd has joined #openstack-infra | 06:24 | |
| StevenK | camunoz: And why is it Invalid API key, rather than No API key? | 06:24 |
| camunoz | that's just our bad :) | 06:25 |
| *** salv-orlando has joined #openstack-infra | 06:30 | |
| *** camunoz has quit IRC | 06:30 | |
| *** sks has quit IRC | 06:33 | |
| *** k4n0 has joined #openstack-infra | 06:33 | |
| *** HeOS has quit IRC | 06:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Create new stackforge project python-watcherclient https://review.openstack.org/185006 | 06:39 |
| *** kaisers1 has joined #openstack-infra | 06:39 | |
| *** amotoki_ has joined #openstack-infra | 06:41 | |
| *** dguerri`away is now known as dguerri | 06:41 | |
| *** Sree_ has joined #openstack-infra | 06:41 | |
| *** Sree_ is now known as Guest29419 | 06:42 | |
| *** Guest60697 has quit IRC | 06:42 | |
| *** e0ne has joined #openstack-infra | 06:42 | |
| *** amotoki_ has quit IRC | 06:45 | |
| *** dguerri is now known as dguerri`away | 06:46 | |
| *** sks has joined #openstack-infra | 06:48 | |
| *** e0ne has quit IRC | 06:49 | |
| *** woodster_ has quit IRC | 06:50 | |
| *** unicell has quit IRC | 06:52 | |
| *** unicell has joined #openstack-infra | 06:52 | |
| *** Guest29419 has quit IRC | 06:54 | |
| *** dtantsur|afk is now known as dtantsur | 06:55 | |
| StevenK | AJaeger: http://paste.openstack.org/show/244145/ | 06:55 |
| *** yamada-h has joined #openstack-infra | 07:00 | |
| *** kaisers2 has joined #openstack-infra | 07:03 | |
| *** otter768 has joined #openstack-infra | 07:03 | |
| *** leopoldj has joined #openstack-infra | 07:04 | |
| *** yamamoto has quit IRC | 07:05 | |
| *** kaisers2 has quit IRC | 07:06 | |
| *** dguerri`away is now known as dguerri | 07:07 | |
| *** hdd has quit IRC | 07:08 | |
| *** markus_z has joined #openstack-infra | 07:08 | |
| *** otter768 has quit IRC | 07:08 | |
| *** mpavone has joined #openstack-infra | 07:08 | |
| *** dguerri is now known as dguerri`away | 07:08 | |
| *** mohitsharma has joined #openstack-infra | 07:10 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/project-config: Switch to using $() for subshells https://review.openstack.org/184907 | 07:11 |
| *** devvesa has joined #openstack-infra | 07:13 | |
| *** unicell1 has joined #openstack-infra | 07:15 | |
| *** unicell has quit IRC | 07:16 | |
| *** achanda has quit IRC | 07:22 | |
| *** yamada-h has quit IRC | 07:23 | |
| *** koolhead17 has quit IRC | 07:24 | |
| *** mrmartin has joined #openstack-infra | 07:28 | |
| *** samuelBartel has quit IRC | 07:29 | |
| frickler | is there some known badness with zuul scheduling? there is a couple of tasks stuck in "queued" for up to 8h (https://review.openstack.org/186599) | 07:29 |
| *** yamada-h has joined #openstack-infra | 07:32 | |
| ttx | frickler: that one looks a bit stuck indeed | 07:32 |
| ttx | and check-tempest-dsvm-multinode-full jobs are a bit stuck too | 07:33 |
| *** dtantsur is now known as dtantsur|brb | 07:35 | |
| *** Ala has joined #openstack-infra | 07:35 | |
| *** unicell has joined #openstack-infra | 07:35 | |
| openstackgerrit | Arnold Bechtoldt proposed openstack-infra/jenkins-job-builder: support com.cloudbees.plugins.credentials properties in authorization matrix https://review.openstack.org/186668 | 07:36 |
| *** unicell1 has quit IRC | 07:36 | |
| *** jistr has joined #openstack-infra | 07:38 | |
| *** hashar has joined #openstack-infra | 07:39 | |
| *** Ritesh has joined #openstack-infra | 07:39 | |
| Ritesh | Hi All, | 07:39 |
| Ritesh | Could you please tell me , where does the tempest logs gets saved | 07:39 |
| Ritesh | I have runned the ./run_tempest.sh in tempest folder | 07:40 |
| Ritesh | I am not able to list of tempest failed test cases | 07:40 |
| Ritesh | like testr | 07:40 |
| *** markvoelker has joined #openstack-infra | 07:40 | |
| *** arxcruz has joined #openstack-infra | 07:41 | |
| *** achanda has joined #openstack-infra | 07:44 | |
| openstackgerrit | Jerome Hourquebie proposed stackforge/python-jenkins: Adds Cloudbees folder plugin support https://review.openstack.org/180185 | 07:45 |
| *** erikmwilson_ has joined #openstack-infra | 07:45 | |
| *** markvoelker has quit IRC | 07:45 | |
| *** mpaolino has joined #openstack-infra | 07:46 | |
| *** jlanoux has joined #openstack-infra | 07:47 | |
| *** erikmwilson_ has quit IRC | 07:50 | |
| *** shardy has joined #openstack-infra | 07:50 | |
| *** ParsectiX has joined #openstack-infra | 07:52 | |
| *** Longgeek has quit IRC | 07:54 | |
| *** mohitsharma has quit IRC | 07:55 | |
| *** sdake_ has joined #openstack-infra | 07:55 | |
| *** stevemar has quit IRC | 07:55 | |
| *** jerryz has quit IRC | 07:56 | |
| *** sdake has quit IRC | 07:59 | |
| *** samuelBartel has joined #openstack-infra | 08:00 | |
| *** HeOS has joined #openstack-infra | 08:01 | |
| *** abregman has joined #openstack-infra | 08:02 | |
| *** dguerri`away is now known as dguerri | 08:04 | |
| *** ddieterly has joined #openstack-infra | 08:09 | |
| *** hashar has quit IRC | 08:10 | |
| *** hashar has joined #openstack-infra | 08:10 | |
| *** hashar is now known as hasharMeeting | 08:11 | |
| *** dims_ has joined #openstack-infra | 08:12 | |
| *** ddieterly has quit IRC | 08:13 | |
| *** nfedotov has joined #openstack-infra | 08:14 | |
| *** kaisers1 has quit IRC | 08:15 | |
| *** soren has joined #openstack-infra | 08:16 | |
| *** sdake_ has quit IRC | 08:17 | |
| electrical | Morning all | 08:17 |
| frickler | ttx: seems to be catching up now, but pretty slowly. looking at the node stats, there is also some constant deletion going on since about 18h, not sure if that might be related | 08:18 |
| *** dims_ has quit IRC | 08:18 | |
| *** yamada-h has quit IRC | 08:18 | |
| *** dizquierdo has joined #openstack-infra | 08:18 | |
| ttx | frickler: yeah, looks like some nodes are rare | 08:19 |
| *** Hal has joined #openstack-infra | 08:23 | |
| *** derekh has joined #openstack-infra | 08:23 | |
| *** Hal is now known as Guest98495 | 08:24 | |
| *** hasharMeeting has quit IRC | 08:24 | |
| *** hashar has joined #openstack-infra | 08:25 | |
| *** ZZelle has quit IRC | 08:27 | |
| *** ZZelle has joined #openstack-infra | 08:28 | |
| *** mleroy has quit IRC | 08:29 | |
| *** mleroy has joined #openstack-infra | 08:30 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Expose Fuel project specs to specs.openstack.org https://review.openstack.org/186675 | 08:30 |
| *** pblaho has joined #openstack-infra | 08:30 | |
| *** achanda has quit IRC | 08:35 | |
| *** fawadkhaliq has quit IRC | 08:36 | |
| *** hrubi has quit IRC | 08:37 | |
| *** btully has quit IRC | 08:38 | |
| *** hrubi has joined #openstack-infra | 08:41 | |
| *** Ritesh has quit IRC | 08:43 | |
| *** hashar_ has joined #openstack-infra | 08:46 | |
| *** hashar has quit IRC | 08:46 | |
| *** fhubik has joined #openstack-infra | 08:47 | |
| *** achanda has joined #openstack-infra | 08:48 | |
| openstackgerrit | Jaume Devesa proposed openstack-infra/project-config: Define MidoNet v2 devstack-gate jobs https://review.openstack.org/181932 | 08:48 |
| *** ParsectiX has quit IRC | 08:53 | |
| *** fijal__ has joined #openstack-infra | 08:58 | |
| *** freerunner has joined #openstack-infra | 08:59 | |
| *** claudiub has quit IRC | 08:59 | |
| *** fhubik is now known as fhubik_afk | 09:01 | |
| *** fijal_ has quit IRC | 09:01 | |
| *** mleroy has quit IRC | 09:02 | |
| *** achanda has quit IRC | 09:02 | |
| *** mleroy has joined #openstack-infra | 09:02 | |
| *** freerunner has quit IRC | 09:03 | |
| *** otter768 has joined #openstack-infra | 09:04 | |
| *** otter768 has quit IRC | 09:09 | |
| *** heyongli has quit IRC | 09:10 | |
| *** freerunner has joined #openstack-infra | 09:10 | |
| *** mpaolino has quit IRC | 09:11 | |
| *** claudiub has joined #openstack-infra | 09:12 | |
| *** fijal__ is now known as fijal | 09:22 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Add pacemaker status to host_info.txt https://review.openstack.org/186690 | 09:28 |
| openstackgerrit | Merged openstack-infra/yaml2ical: Add timestamp to the template variables https://review.openstack.org/186390 | 09:29 |
| *** markvoelker has joined #openstack-infra | 09:29 | |
| *** mpaolino has joined #openstack-infra | 09:30 | |
| *** mpaolino has quit IRC | 09:31 | |
| *** freerunner has quit IRC | 09:31 | |
| *** freerunner has joined #openstack-infra | 09:32 | |
| *** markvoelker has quit IRC | 09:34 | |
| *** fhubik_afk is now known as fhubik | 09:34 | |
| *** hashar_ is now known as hashar | 09:36 | |
| *** fhubik has quit IRC | 09:41 | |
| *** fhubik has joined #openstack-infra | 09:41 | |
| *** ParsectiX has joined #openstack-infra | 09:44 | |
| *** erikmwilson_ has joined #openstack-infra | 09:46 | |
| *** fawadkhaliq has joined #openstack-infra | 09:49 | |
| *** erikmwilson_ has quit IRC | 09:51 | |
| *** dims_ has joined #openstack-infra | 09:51 | |
| *** ParsectiX has quit IRC | 09:55 | |
| *** dizquierdo has quit IRC | 09:57 | |
| *** ildikov has joined #openstack-infra | 10:04 | |
| *** jyuso1 has joined #openstack-infra | 10:06 | |
| *** freerunner has quit IRC | 10:07 | |
| openstackgerrit | Richard Pijnenburg proposed openstack-infra/jenkins-job-builder: Added slack support https://review.openstack.org/131404 | 10:10 |
| *** ddieterly has joined #openstack-infra | 10:10 | |
| *** afazekas has joined #openstack-infra | 10:12 | |
| *** johnthetubaguy is now known as MrTubaNotSousa | 10:14 | |
| *** ddieterly has quit IRC | 10:14 | |
| *** MrTubaNotSousa is now known as JohnTubaNotSousa | 10:15 | |
| *** btully has joined #openstack-infra | 10:17 | |
| *** amotoki_ has joined #openstack-infra | 10:17 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings: Add meeting_id and agenda_url support https://review.openstack.org/186184 | 10:18 |
| *** btully has quit IRC | 10:22 | |
| *** amotoki_ has quit IRC | 10:22 | |
| *** mjordan has joined #openstack-infra | 10:23 | |
| *** baohua has joined #openstack-infra | 10:24 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings: Add meeting_id and agenda_url support https://review.openstack.org/186184 | 10:26 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings: Add index page generation timestamp https://review.openstack.org/186701 | 10:26 |
| *** JohnTubaNotSousa is now known as johnthetubaguy | 10:27 | |
| *** achanda has joined #openstack-infra | 10:28 | |
| *** samueldmq has joined #openstack-infra | 10:30 | |
| samueldmq | morning | 10:30 |
| katyafervent | Hi all! Please help us with fixing stable/juno https://review.openstack.org/#/c/184809/ | 10:30 |
| katyafervent | This fix is waiting for a merge :) | 10:30 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add experimental Rally jobs for Manila https://review.openstack.org/186702 | 10:30 |
| *** ihrachyshka has joined #openstack-infra | 10:33 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: remove cookbook-pacemaker from infra https://review.openstack.org/178298 | 10:33 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add experimental Rally jobs for Manila https://review.openstack.org/186702 | 10:35 |
| samueldmq | katyafervent, so you need #184196 to fix the gate, and you can't fix the gate because it doesn't succeed against the patch which is fixing it :) | 10:37 |
| mordred | AJaeger: looks like it's there and imported | 10:38 |
| samueldmq | katyafervent, if I understand correctly, you are then disabling the job as voting, fix and make it voting again, right ? | 10:38 |
| *** dtantsur|brb is now known as dtantsur | 10:38 | |
| katyafervent | yes) And tox is not able to install venv from stable/juno branch | 10:38 |
| katyafervent | samueldmq, that is correct, it should be non-voting only for stable/juno and only for 1-2 days | 10:39 |
| samueldmq | mordred, good morning | 10:39 |
| *** pc_m has joined #openstack-infra | 10:39 | |
| samueldmq | katyafervent, k I will take a look; though I am not core on project-config | 10:40 |
| *** fawadkhaliq has quit IRC | 10:42 | |
| AJaeger | mordred: I checked http://git.openstack.org/cgit - and don't see it. | 10:44 |
| katyafervent | samueldmq, thank you! :) but you should know who is core unlike me | 10:44 |
| AJaeger | mordred: oh, found it... | 10:44 |
| mordred | phew | 10:44 |
| AJaeger | mordred: not sure why I didn't see it earlier ;( Sorry for the false alarm :( | 10:45 |
| mordred | AJaeger: no worries! I'm just glad there isn't a problem | 10:45 |
| electrical | AJaeger: mordred morning guys | 10:45 |
| AJaeger | morning, electrical ! | 10:46 |
| AJaeger | katyafervent: you can easily figure out who's core... | 10:46 |
| AJaeger | katyafervent: do you wnat to know how? | 10:46 |
| electrical | time to do an other review round for JJB open changes. | 10:46 |
| *** hichihara has quit IRC | 10:46 | |
| samueldmq | AJaeger, lp ? | 10:47 |
| AJaeger | samueldmq: just open a review for project-config like https://review.openstack.org/186702 | 10:47 |
| AJaeger | click on the name " openstack-infra/project-config" | 10:47 |
| AJaeger | click on "Access" | 10:48 |
| AJaeger | you see the ACL - click on project-config-core and you see the cores | 10:48 |
| dtantsur | hi folks! any ETA on etherpad recovering from the summit? | 10:48 |
| AJaeger | https://review.openstack.org/#/admin/groups/459,members | 10:48 |
| AJaeger | brb | 10:48 |
| samueldmq | AJaeger, hmm, and groups can be contained by other groups | 10:49 |
| dtantsur | I'd like to know when we'll be able to save changes again.. | 10:49 |
| samueldmq | AJaeger, I wonder if that could be good to have on keystone :) | 10:49 |
| *** witek has joined #openstack-infra | 10:49 | |
| *** mjordan has quit IRC | 10:51 | |
| *** dims_ has quit IRC | 10:52 | |
| samueldmq | katyafervent, 2015-05-14 17:55:00.049 | ValueError: git history requires a target version of pbr.version.SemanticVersion(2014.2.3), but target version is pbr.version.SemanticVersion(2014.2.1) | 10:53 |
| *** amotoki has quit IRC | 10:53 | |
| *** ParsectiX has joined #openstack-infra | 11:04 | |
| *** jyuso1 has quit IRC | 11:05 | |
| *** otter768 has joined #openstack-infra | 11:05 | |
| *** freerunner has joined #openstack-infra | 11:05 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add pacemaker status to host_info.txt https://review.openstack.org/186690 | 11:06 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: Rename stackforge/chef* to openstack/chef* https://review.openstack.org/186532 | 11:07 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: remove cookbook-pacemaker from infra https://review.openstack.org/178298 | 11:07 |
| *** nfedotov has quit IRC | 11:07 | |
| *** jyuso1 has joined #openstack-infra | 11:08 | |
| *** AzherKhan has quit IRC | 11:09 | |
| openstackgerrit | Dawid Malinowski proposed openstack-infra/jenkins-job-builder: Extend support for join-trigger https://review.openstack.org/186176 | 11:09 |
| *** dizquierdo has joined #openstack-infra | 11:09 | |
| *** otter768 has quit IRC | 11:10 | |
| *** ddieterly has joined #openstack-infra | 11:11 | |
| *** juggler has quit IRC | 11:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/186650 | 11:14 |
| *** juggler has joined #openstack-infra | 11:14 | |
| *** ddieterly has quit IRC | 11:16 | |
| *** teran has joined #openstack-infra | 11:16 | |
| *** pasquier-s has quit IRC | 11:17 | |
| *** SergK has quit IRC | 11:17 | |
| *** markvoelker has joined #openstack-infra | 11:18 | |
| *** panda has quit IRC | 11:21 | |
| *** panda has joined #openstack-infra | 11:21 | |
| *** markvoelker has quit IRC | 11:22 | |
| *** mattt has joined #openstack-infra | 11:23 | |
| mattt | lifeless: ping | 11:24 |
| *** ParsectiX has quit IRC | 11:28 | |
| *** SergK has joined #openstack-infra | 11:30 | |
| *** pasquier-s has joined #openstack-infra | 11:31 | |
| ttx | fungi: when you are up, seems to be some node attrition right now | 11:37 |
| *** baoli has joined #openstack-infra | 11:39 | |
| *** baoli_ has joined #openstack-infra | 11:40 | |
| *** baoli_ has quit IRC | 11:40 | |
| openstackgerrit | Dawid Malinowski proposed openstack-infra/jenkins-job-builder: Extend support for join-trigger https://review.openstack.org/186176 | 11:40 |
| *** mjordan has joined #openstack-infra | 11:41 | |
| mordred | ttx: oh yeah? | 11:42 |
| *** fawadkhaliq has joined #openstack-infra | 11:43 | |
| ttx | mordred: long backlog on check queue, seem to be max 300 test nodes right now | 11:43 |
| frickler | mordred: also constantly large number of nodes being deleted for the last 12 hours or so | 11:44 |
| mordred | oh goodie | 11:44 |
| ttx | Also sounds like check-tempest-dsvm-multinode-full (non-voting) is delaying nova tests by a handful of hours | 11:44 |
| *** baoli has quit IRC | 11:44 | |
| ttx | since it takes about 5 hours to get a node for that | 11:44 |
| *** aysyanne has joined #openstack-infra | 11:44 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Correct an error in a meeting description. https://review.openstack.org/186075 | 11:45 |
| mordred | well, we started seeing a whole new error yesterday afternoon - first guess is that it has stuck around and gotten worse | 11:45 |
| *** ParsectiX has joined #openstack-infra | 11:45 | |
| *** fhubik is now known as fhubik_afk | 11:46 | |
| *** erikmwilson_ has joined #openstack-infra | 11:47 | |
| *** ZZelle_ has joined #openstack-infra | 11:48 | |
| *** fawadkhaliq has quit IRC | 11:48 | |
| *** amotoki has joined #openstack-infra | 11:49 | |
| *** freerunner has quit IRC | 11:49 | |
| *** freerunner has joined #openstack-infra | 11:49 | |
| *** dims_ has joined #openstack-infra | 11:50 | |
| *** erikmwilson_ has quit IRC | 11:52 | |
| *** baoli has joined #openstack-infra | 11:52 | |
| *** nfedotov has joined #openstack-infra | 11:53 | |
| *** amotoki has quit IRC | 11:54 | |
| *** mwhagedorn has joined #openstack-infra | 11:56 | |
| *** baoli has quit IRC | 11:57 | |
| *** baoli has joined #openstack-infra | 11:58 | |
| *** ldnunes has joined #openstack-infra | 11:59 | |
| *** markvoelker has joined #openstack-infra | 11:59 | |
| *** markus_z has quit IRC | 12:00 | |
| *** doug-fish has joined #openstack-infra | 12:00 | |
| *** dprince has joined #openstack-infra | 12:01 | |
| *** k4n0 has quit IRC | 12:01 | |
| *** bhunter71 has joined #openstack-infra | 12:04 | |
| *** markus_z has joined #openstack-infra | 12:04 | |
| *** baoli_ has joined #openstack-infra | 12:04 | |
| *** btully has joined #openstack-infra | 12:05 | |
| *** achanda has quit IRC | 12:07 | |
| *** baoli has quit IRC | 12:07 | |
| *** btully has quit IRC | 12:10 | |
| *** ddieterly has joined #openstack-infra | 12:12 | |
| *** marcusvrn has joined #openstack-infra | 12:14 | |
| *** miqui has quit IRC | 12:15 | |
| *** fhubik_afk is now known as fhubik | 12:15 | |
| *** mpaolino has joined #openstack-infra | 12:17 | |
| *** ddieterly has quit IRC | 12:17 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add experimental Rally jobs for Manila https://review.openstack.org/186702 | 12:18 |
| *** weshay has quit IRC | 12:24 | |
| *** kgiusti has joined #openstack-infra | 12:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Debug new BadRequest error https://review.openstack.org/186744 | 12:28 |
| mordred | fungi: ^^ I'd like to apply that and restart nodepool | 12:28 |
| *** tiswanso has joined #openstack-infra | 12:29 | |
| *** fhubik is now known as fhubik_afk | 12:30 | |
| *** tiswanso has quit IRC | 12:30 | |
| *** fijal has quit IRC | 12:30 | |
| *** arxcruz has quit IRC | 12:31 | |
| mordred | OverLimit: This request was rate-limited. (HTTP 413) | 12:31 |
| *** tiswanso has joined #openstack-infra | 12:31 | |
| mordred | we're also hitting rate limits | 12:31 |
| *** fhubik_afk is now known as fhubik | 12:31 | |
| *** arxcruz has joined #openstack-infra | 12:33 | |
| frickler | mordred: where can one see these errors? (just being curious) | 12:34 |
| *** weshay has joined #openstack-infra | 12:34 | |
| mordred | frickler: unfortunately you can't - they're in a debug log and we're still not convinced that openstack doesn't leak credentials into it, so we haven't started publishing it | 12:35 |
| *** ayoung has joined #openstack-infra | 12:36 | |
| frickler | mordred: ah, too bad, thx | 12:37 |
| mordred | frickler: yah - I'm not happy about it - would really prefer to be able to have folks help troubleshoot things like this | 12:37 |
| mordred | btw- for anyone following along ... this: BadRequest: Can not find requested image (HTTP 400) (Request-ID: req-6aac123d-dbb5-4efd-bc39-bdb7da14b05a) | 12:38 |
| mordred | is not the friendliest exception in the world | 12:38 |
| *** jsavak has joined #openstack-infra | 12:38 | |
| *** fijal has joined #openstack-infra | 12:39 | |
| *** freerunner has quit IRC | 12:39 | |
| *** bswartz has quit IRC | 12:39 | |
| mordred | fungi, clarkb, jeblair, jhesketh, pleia2, SergeyLukjanov: ^^ I've applied the patch above in production to try to get some better debugging. I've also manually lowered the API rate limit on HP as we were hitting rate limit errors and I didn't want to troubleshoot 2 things at once | 12:39 |
| *** yamamoto has joined #openstack-infra | 12:40 | |
| *** marcusvrn1 has joined #openstack-infra | 12:40 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Update paths to hooks in manila tempest job https://review.openstack.org/186749 | 12:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Debug new BadRequest error https://review.openstack.org/186744 | 12:44 |
| *** marcusvrn has quit IRC | 12:44 | |
| *** ddieterly has joined #openstack-infra | 12:44 | |
| *** miqui has joined #openstack-infra | 12:48 | |
| *** marcusvrn1 has quit IRC | 12:48 | |
| *** marcusvrn has joined #openstack-infra | 12:48 | |
| *** ddieterly has quit IRC | 12:48 | |
| *** dpaterson has joined #openstack-infra | 12:49 | |
| *** amotoki has joined #openstack-infra | 12:49 | |
| *** woodster_ has joined #openstack-infra | 12:50 | |
| *** sks has quit IRC | 12:51 | |
| mescanef | hello.:-) there is a review (https://review.openstack.org/#/c/186675/) regarding "exposing fuel project spec to specs.openstack.org", but the fuel-specs is on stackforge | 12:51 |
| bookwar | hi, everyone, i am interested about this one: https://review.openstack.org/#/c/186675/ - are there technical obstacles to publish stackforge/*-specs to spec.o.o or organizational ones? can we maybe create a specs.stackforge.org or smth like this? | 12:51 |
| bookwar | oops :) | 12:51 |
| mescanef | bookwar: :-) | 12:51 |
| mordred | fungi: ok. I have pulled the params from one of the ones where we get "can't find image" and have verified that it's a valid image uuid and that I _can_ submit a request to build a server with it | 12:53 |
| *** yamamoto has quit IRC | 12:53 | |
| *** dguitarbite has quit IRC | 12:54 | |
| *** amotoki has quit IRC | 12:54 | |
| *** radez_g0n3 is now known as radez | 12:54 | |
| *** marcusvrn has quit IRC | 12:55 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:01 | |
| *** Somay has joined #openstack-infra | 13:03 | |
| *** otter768 has joined #openstack-infra | 13:06 | |
| Somay | Any JJB developers, could you review the following patch - https://review.openstack.org/#/c/185445/ zaro mgagne hashar | 13:06 |
| *** freerunner has joined #openstack-infra | 13:07 | |
| *** marcusvrn has joined #openstack-infra | 13:08 | |
| *** otter768 has quit IRC | 13:11 | |
| *** xyang1 has joined #openstack-infra | 13:11 | |
| fungi | mordred: catching up now | 13:12 |
| mordred | fungi: morning! | 13:13 |
| mordred | fungi: happy to have more eyeballs | 13:13 |
| mordred | fungi: I'm trying rebuilding one of the images on the off chance that something about when we built images last did something evil | 13:14 |
| *** mriedem has joined #openstack-infra | 13:14 | |
| electrical | Morning Somay | 13:14 |
| *** achanda has joined #openstack-infra | 13:14 | |
| Somay | morning electrical :) | 13:14 |
| mordred | fungi: main thing I've been able to verify is that the error we're getting is nova telling us that it had an error requesting things about the image from glance | 13:14 |
| mrmartin | re | 13:14 |
| *** fawadkhaliq has joined #openstack-infra | 13:14 | |
| fungi | mordred: well, looking at the graph it seems this probably began when nodepool was restarted around 2300z yesterday for the new patches that landed? | 13:14 |
| mordred | fungi: so it's not that we've submitted an invalid image id to nova, or a malformed request | 13:14 |
| *** akshai has joined #openstack-infra | 13:15 | |
| mordred | fungi: no - it turns out it was happening before that - clarkb and I noticed the errors aroudn the restart | 13:15 |
| fungi | oh, got it | 13:15 |
| mordred | fungi: but I grepped old logs and see the error before hand | 13:15 |
| mordred | fungi: I'm wondering if we've reached an image size that's too big for nova to deal with and it's vomiting | 13:16 |
| mordred | or vomiting some times | 13:16 |
| mordred | or something | 13:16 |
| mordred | also - we're hitting this on both clouds | 13:16 |
| mordred | and it's intermittend | 13:16 |
| mordred | and it's intermittent | 13:16 |
| fungi | is it limited to specific images? | 13:16 |
| electrical | Somay: could you code check https://review.openstack.org/#/c/131404/ ? | 13:17 |
| mordred | fungi: it has happened on 45 different images since I restarted to add the debugging | 13:17 |
| mordred | grep 'Task: image:' debug.log | awk '{print $6}' | sort | uniq | wc -l | 13:17 |
| *** mpaolino has quit IRC | 13:17 | |
| fungi | thinking through the way nova works... it pulls the image to the compute node's filesystem normally and caches it locally, yeah? | 13:17 |
| mordred | yes | 13:17 |
| Somay | electrical: Sure, I’ll have a look :) | 13:17 |
| electrical | thnx | 13:17 |
| mordred | fungi: but it should not do that at api create time | 13:17 |
| mordred | I thnik api create time all it does it ping glance to make sure the requested image exists at all? | 13:18 |
| fungi | oh, this is not an error from the boot call then | 13:18 |
| mordred | it is an error from the boot call | 13:18 |
| mordred | it's _not_ an error in the boot sequence | 13:18 |
| mordred | we get it as an error when we're trying to submit the request to nova that nova spin up a vm | 13:18 |
| mordred | apparently nova api is talking to glance API before returning | 13:19 |
| fungi | okay, so this is well before a compute node would be assigned and then the image pulled into its cache if necessary | 13:19 |
| *** fawadkhaliq has quit IRC | 13:19 | |
| mordred | yah. at least, that's my understanding | 13:19 |
| *** mattt has left #openstack-infra | 13:21 | |
| fungi | indeed, so we were getting "Waiting for server None for node id: ..." in the debug log it looks like | 13:22 |
| fungi | prior to your debugging patch | 13:22 |
| *** ddieterly has joined #openstack-infra | 13:22 | |
| fungi | wondering if we have a lot of None nodes in the db now | 13:23 |
| *** dboik has joined #openstack-infra | 13:24 | |
| mrmartin | nibalizer: are u here? | 13:24 |
| fungi | mordred: oh, actually looks like maybe that was the result of some prior debugging of yours | 13:25 |
| katyafervent | hi fungi ! Please, take a look at https://review.openstack.org/#/c/184809/ | 13:26 |
| fungi | since it doesn't start up until 12:43z | 13:26 |
| *** Somay has quit IRC | 13:26 | |
| fungi | katyafervent: perhaps later when we're not trying to fix our ci | 13:26 |
| katyafervent | Murano installation with pbr is broken and blocked until this fix would be merged | 13:26 |
| *** mtanino has joined #openstack-infra | 13:26 | |
| katyafervent | Ok, thank you | 13:26 |
| fungi | not much is merging for anyone at all right now | 13:26 |
| mordred | fungi: I had two abortive attempts at debugging | 13:26 |
| fungi | mordred: no worries, just led me down a confusing path for a bit when excavating logs | 13:27 |
| electrical | fungi: excavating logs? not using the ELK stack then ? ;-) lol | 13:27 |
| fungi | electrical: using vi. it's like logstash except... not | 13:27 |
| electrical | hehe ;-) | 13:28 |
| mordred | fungi: btw - I'm asking the HP noc to trace some request ids for us | 13:29 |
| mordred | to see what we can learn | 13:29 |
| *** annegentle has joined #openstack-infra | 13:29 | |
| *** mpaolino has joined #openstack-infra | 13:29 | |
| *** koolhead17 has joined #openstack-infra | 13:31 | |
| fungi | yeah, we have the request id but not anything to tie it to a specific image, provider, anything | 13:32 |
| fungi | oh, wait, that's a multiline log entry | 13:32 |
| fungi | heh. we have lots of detail! | 13:32 |
| mordred | yes! _plenty_ of detail | 13:33 |
| fungi | out of curiosity, i'm looking to see what our alien-image-list looks like right now | 13:34 |
| *** Somay has joined #openstack-infra | 13:34 | |
| fungi | actually not a bunch, so unlikely to be at all related to having an excess count of images stored anywhere (looks like 29 total across all providers right now) | 13:34 |
| fungi | 45 distinct image uuids referenced for that error so doesn't seem to be particular to a single image | 13:36 |
| *** puranamr has joined #openstack-infra | 13:37 | |
| *** dguitarbite has joined #openstack-infra | 13:38 | |
| *** abregman has quit IRC | 13:38 | |
| *** pblaho has quit IRC | 13:38 | |
| ttx | mordred: was wondering... should we have a stable/kilo branch for openstack/openstack ? | 13:38 |
| ttx | I'm writing the "let's stop making stable point release" thread and was trying to anticipate the "how to know what was tested together" question | 13:39 |
| *** Somay has quit IRC | 13:39 | |
| ttx | (as if we actually did test the tagged versions together :) | 13:39 |
| mordred | ttx: yah. we probably should | 13:40 |
| *** puranamr has quit IRC | 13:40 | |
| *** mjordan has quit IRC | 13:42 | |
| ttx | not sure how the magic works there. Will it automatically push a commit there if the branch exists ? | 13:42 |
| ttx | (if we do, can't hurt to also create s stable/juno there since it's missing as well | 13:43 |
| ttx | mordred: ^ | 13:43 |
| fungi | mordred: picking the highest-failing image uuid from the current debug log, i've confirmed that it is indeed shown active in nova image-list so it's not like it's not there at all. that's a good sign at least | 13:43 |
| *** ayoung has quit IRC | 13:43 | |
| mordred | ttx: yah - we need to basically make a branch, and then make a commit to that branch changing the .gitmodules references from master to stable/kilo | 13:43 |
| mordred | fungi: yah | 13:43 |
| *** dguitarbite has quit IRC | 13:44 | |
| mordred | fungi: earlier I also confirmed that I could boot off of one of the images that was failing | 13:44 |
| derekh | If anybody has a chance could I get some eyes on a patch to move tripleo ci to F21 https://review.openstack.org/#/c/183513/ | 13:44 |
| ttx | mordred: are you on it ? No idea where we should be branching *from* (or if it does actually mean anything in this strange repo) | 13:44 |
| *** bswartz has joined #openstack-infra | 13:44 | |
| *** annegent_ has joined #openstack-infra | 13:44 | |
| *** annegentle has quit IRC | 13:45 | |
| mordred | ttx: I'll put it on my TDL | 13:45 |
| *** gokrokve has joined #openstack-infra | 13:45 | |
| *** gokrokve has quit IRC | 13:45 | |
| *** gokrokve has joined #openstack-infra | 13:46 | |
| *** koolhead17 has quit IRC | 13:47 | |
| *** med_ has joined #openstack-infra | 13:48 | |
| fungi | and yes, this error does seem to cut relatively proportionally across all providers/regions/labels | 13:49 |
| fungi | including images which are dibified and those which are snapshotty | 13:49 |
| fungi | so, yeah... while the fact that this is fairly evenly spread across our resources suggests it should be a bug in nodepoold, i have a really hard time seeing how it could be | 13:54 |
| fungi | unless it's lying to us about the api calls it's making? | 13:54 |
| mordred | fungi: I did trace the error string back into nova | 13:55 |
| mordred | fungi: and it is a fairly unique string, even though it's a bad one | 13:55 |
| mordred | fungi: and really is only returned by the nova api | 13:56 |
| *** mattt has joined #openstack-infra | 13:56 | |
| *** annegent_ has quit IRC | 13:57 | |
| *** fhubik has quit IRC | 13:57 | |
| fungi | the only other explanation i have is that hpcloud and rackspace are continuously deploying nova and this is a recently introduced bug | 13:57 |
| *** fawadkhaliq has joined #openstack-infra | 13:57 | |
| pabelanger | morning | 14:00 |
| fungi | so... the first instance of this i can find in the logs is 2015-05-28 17:57:30,712 | 14:01 |
| *** baohua has quit IRC | 14:01 | |
| pabelanger | fungi, was reading scroll back from last night, do you manually build dibs? Or is it a crontab for the image-update? | 14:01 |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor https://review.openstack.org/181873 | 14:01 |
| fungi | pabelanger: timer entry in nodepool.yaml | 14:01 |
| pabelanger | cool | 14:02 |
| fungi | i'm going to see if i can correlate that to any changes we've made, and also see if it started in one provider and then later the other one joined in | 14:02 |
| *** krtaylor has quit IRC | 14:02 | |
| *** ayoung has joined #openstack-infra | 14:03 | |
| fungi | so... at 17:32z we merged "Set image-type to vhd for rax nodepool providers" | 14:04 |
| fungi | but i don't think we created any new images since then | 14:04 |
| *** emagana has quit IRC | 14:04 | |
| *** mtanino has quit IRC | 14:05 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 14:06 | |
| *** afazekas has quit IRC | 14:06 | |
| mordred | fungi: that seems like a weird thing to have triggered this | 14:08 |
| mordred | fungi: most of the images seem to be from 45ish hours ago | 14:08 |
| *** nelsnelson has joined #openstack-infra | 14:08 | |
| fungi | yep | 14:09 |
| fungi | just pointing out i don't see any changes likely to have coincided with the start of this | 14:09 |
| *** achanda has quit IRC | 14:10 | |
| *** annegentle has joined #openstack-infra | 14:10 | |
| fungi | i also have a friend at rackspace who says he just finished catching their nova fork branch up to merges from upstream master yesterday. i don't know how quickly it gets to deployed from there though | 14:11 |
| *** btully has joined #openstack-infra | 14:12 | |
| *** salv-orlando has quit IRC | 14:12 | |
| *** Sukhdev has joined #openstack-infra | 14:13 | |
| *** cody-somerville has quit IRC | 14:13 | |
| *** peristeri has joined #openstack-infra | 14:13 | |
| *** krtaylor has joined #openstack-infra | 14:13 | |
| fungi | so, interestingly... we saw a short burst of 22 of these in hpcloud yesterday from 17:57:30-17:58:03 and then nothing again until 22:23:19 when it started appearing in both rax and hpcloud | 14:15 |
| fungi | from that point on it's been basically continuous | 14:15 |
| *** dims_ has quit IRC | 14:16 | |
| *** zz_jgrimm is now known as jgrimm | 14:17 | |
| fungi | the timing is suspiciously simultaneous for both providers that i don't see how it could be a new nova bug suddenly cropping up in both of them | 14:17 |
| *** leopoldj has quit IRC | 14:17 | |
| *** mpaolino has quit IRC | 14:18 | |
| fungi | there's no gap in the debug log itself either, just a ~4.5 hour gap in those particular errors | 14:19 |
| *** mrunge has quit IRC | 14:19 | |
| *** wenlock has joined #openstack-infra | 14:21 | |
| *** timcline has joined #openstack-infra | 14:21 | |
| fungi | interesting... nodepool reloads its configuration every 20 seconds according to its debug log | 14:21 |
| *** vaidy_ has joined #openstack-infra | 14:25 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:25 | |
| *** jistr has quit IRC | 14:26 | |
| fungi | the initial burst of these errors does not coincide with when puppet updated the nodepool configuration. it was ~15 minutes prior (nodepoold saw the config update at 18:08:16) | 14:26 |
| fungi | _however_ when the errors started back up later, it was 30 seconds after we restarted nodepoold at 22:22:55 | 14:28 |
| fungi | i have a feeling if we roll back whatever changes took effect in the last restart this will subside | 14:29 |
| *** ayoung has quit IRC | 14:29 | |
| fungi | that timing is just too close to be a coincidence | 14:29 |
| *** olaph has quit IRC | 14:29 | |
| jeblair | o/ | 14:30 |
| *** cody-somerville has joined #openstack-infra | 14:30 | |
| mordred | fungi: last things I think had to do with image properties | 14:30 |
| *** olaph has joined #openstack-infra | 14:30 | |
| mordred | for dib images | 14:30 |
| *** tonytan4ever has joined #openstack-infra | 14:31 | |
| clarkb | but only on rax | 14:31 |
| *** mpaolino has joined #openstack-infra | 14:31 | |
| *** fijal has quit IRC | 14:32 | |
| clarkb | and only on ubuntu-trusty. | 14:32 |
| fungi | clarkb: are you talking about configuration or nodepool source code? | 14:32 |
| clarkb | config | 14:33 |
| fungi | config updates don't _seem_ to correlate with any of this | 14:33 |
| fungi | however the nodepoold restart at 22:22:55 was immediately followed with these errors starting to appear in both providers at 22:23:19 | 14:33 |
| clarkb | oh I am confused by mordreds comment then. The restart was to pick up changes to properly reload clouds.yaml config iirc | 14:33 |
| fungi | aha, yes that could be clouds.yaml then since nodepoold wasn't automatically picking up those | 14:34 |
| *** TravT has quit IRC | 14:34 | |
| fungi | i was simply correlating to updates for nodepool.yaml which weren't apparently related timing wise | 14:34 |
| *** freerunner has quit IRC | 14:35 | |
| * mordred has to AFK | 14:35 | |
| *** esker has joined #openstack-infra | 14:35 | |
| clarkb | ya I dont think clouds.yaml was updated either, just nodepools ability to make new managers when clouds.yaml does update | 14:36 |
| fungi | i can't account for the very short (30-second) burst of these errors just prior to 1800, but what we're seeing now definitely started immediately after the nodepoold restart | 14:36 |
| jeblair | mordred's debug patch doesn't seem to work | 14:37 |
| clarkb | maybe the related shade update introduced a client bug? but we dont use shade for launching nodes yet, we just get the novaclient from shade and use that | 14:37 |
| fungi | jeblair: doesn't seem to work how? note that it's multi-line debug entries | 14:38 |
| jeblair | it looks like he's evolving the patch on disk | 14:38 |
| fungi | oh | 14:38 |
| *** amotoki has joined #openstack-infra | 14:38 | |
| jeblair | fungi: yeah, the version in gerrit has a programming error | 14:38 |
| jeblair | the version on disk looks like it should fix that but isn't actually outputting anything | 14:38 |
| *** freerunner has joined #openstack-infra | 14:39 | |
| jeblair | so either it has an error, or self.args.items is really empty | 14:39 |
| jeblair | - for (key, value) in self.args: | 14:39 |
| jeblair | + for (key, value) in self.args.items(): | 14:39 |
| jeblair | is the diff between the two | 14:39 |
| *** stevemar has joined #openstack-infra | 14:39 | |
| fungi | hrm... i'm definitely seeing request ids, image uuids, et cetera in the debug log | 14:39 |
| jeblair | oh, am i missing those... | 14:39 |
| jeblair | oh i do see them now, sorry | 14:40 |
| *** s0nea has quit IRC | 14:40 | |
| fungi | yeah, i think for this to be a permanent debug improvement those multi-line entries need to be better correlated somehow (a common pre4fix or something) | 14:41 |
| jeblair | http://paste.openstack.org/show/244970/ | 14:41 |
| jeblair | so there's an example ^ | 14:41 |
| openstackgerrit | Merged openstack-infra/project-config: Make 'gate-murano-devstack-dsvm' non-voting https://review.openstack.org/184809 | 14:42 |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| fungi | right | 14:42 |
| jeblair | the error has never occurred in dfw | 14:42 |
| fungi | weird. i missed that | 14:43 |
| *** dangers_away is now known as dangers | 14:43 | |
| *** mjordan has joined #openstack-infra | 14:43 | |
| *** amotoki has quit IRC | 14:43 | |
| jeblair | it has occurred in hpcloud-* (but that's all one region (however, it is multiple AZs and that's not logged) | 14:43 |
| *** ayoung has joined #openstack-infra | 14:44 | |
| jeblair | is it possible that we have only just now started requesting servers in rax-ord after having uploaded an image to rax-dfw? (eg, previously we were still requesting snapshot images in ord) | 14:44 |
| *** sdake has joined #openstack-infra | 14:46 | |
| jeblair | oh, the azs are logged in the key/value portion of the logs | 14:46 |
| fungi | maybe... keep in mind that i deleted all yesterday's images so we've been booting from images from day-before-yesterday | 14:46 |
| *** yamamoto has quit IRC | 14:47 | |
| jeblair | and that's still the case because today's images aren't ready yet, right? | 14:47 |
| *** jistr has joined #openstack-infra | 14:47 | |
| fungi | yep | 14:47 |
| *** ajmiller has joined #openstack-infra | 14:48 | |
| fungi | so it's _possible_ this could subside as soon as new images get uploaded today, i suppose | 14:48 |
| *** fijal has joined #openstack-infra | 14:48 | |
| jeblair | oh, and it's affecting snapshot images in rax | 14:48 |
| fungi | though... we're also definitely seeing this for snapshot-only labels | 14:49 |
| jeblair | i just assumed it was a dib image | 14:49 |
| fungi | yeah, like bare-.* are also impacted equally and they're all still snapshot | 14:49 |
| fungi | mordred: i take it you've gotten nowhere with the hp noc tracing an example request id | 14:50 |
| fungi | oh, right, mordred said he had to disappear for a bit | 14:51 |
| jeblair | oh, he started that? lemme see if i can followup | 14:51 |
| fungi | yeah, an hour or two ago | 14:51 |
| *** amotoki has joined #openstack-infra | 14:52 | |
| *** armax has joined #openstack-infra | 14:57 | |
| spzala | Hello, we want to start a new IRC channel for 'heat-translator' project considering growing team from different time zones. Can someone please help with the set up? | 14:58 |
| *** esker has quit IRC | 14:59 | |
| *** s0nea has joined #openstack-infra | 14:59 | |
| *** annegentle has quit IRC | 14:59 | |
| *** esker has joined #openstack-infra | 15:00 | |
| *** achanda has joined #openstack-infra | 15:00 | |
| jeblair | spzala: you can run the commands listed here: http://docs.openstack.org/infra/system-config/irc.html#access and then add them to the config files in the openstack-infra/project config repo. consider using "#openstack-" as a prefix to the channel name | 15:00 |
| *** mjordan has quit IRC | 15:00 | |
| openstackgerrit | Michal Vanco proposed openstack-infra/jenkins-job-builder: Support for image gallery publisher https://review.openstack.org/175747 | 15:00 |
| spzala | jeblair: thank you much!! Sounds good. | 15:01 |
| *** emagana has joined #openstack-infra | 15:02 | |
| openstackgerrit | Merged openstack/requirements: Fix requirement conflict with flake8. https://review.openstack.org/183930 | 15:04 |
| *** hdd has joined #openstack-infra | 15:05 | |
| *** fijal has quit IRC | 15:06 | |
| openstackgerrit | Merged openstack/requirements: Add pbr to synchronised projects. https://review.openstack.org/184293 | 15:06 |
| *** fijal has joined #openstack-infra | 15:07 | |
| *** otter768 has joined #openstack-infra | 15:07 | |
| jeblair | when i do an alien list of rax-iad i see nodes from dfw and ord | 15:07 |
| jeblair | this is normal for hpcloud because of the pseudo regions, but i don't think it's normal for rax | 15:07 |
| fungi | oh! | 15:08 |
| *** mrmartin has quit IRC | 15:08 | |
| fungi | yeah, that i think points to a real problem | 15:08 |
| *** soren has quit IRC | 15:09 | |
| fungi | i wonder if i can install nodepool from an earlier ref into a venv and bisect it that way | 15:09 |
| fungi | playing with that possibility now | 15:09 |
| jeblair | (when i nova list with the iad creds on puppetmaster, i only see iad hosts as expected) | 15:10 |
| clarkb | jeblair: possible that shade is being too smart for us there (though again we should still just be using the novaclient made by shade so probably not) | 15:10 |
| jeblair | so i'm wondering if something has gone awry with the "use novaclient from shade" idea | 15:10 |
| stevemar | o/ | 15:11 |
| fungi | yeah, like we're reusing a client object for the wrong provider | 15:11 |
| clarkb | fungi: oh and if that was the case maybe we are giving a valid image uuid to the wrong region | 15:11 |
| *** marekd has joined #openstack-infra | 15:11 | |
| jeblair | that would definitely cause an image not to be found in rax (but hpcloud? -- unless we used a rax provider to build an hpcloud server) | 15:11 |
| *** otter768 has quit IRC | 15:11 | |
| stevemar | whom can i bug about creating a new subproject under keystone (morgan was already cool with it) | 15:11 |
| clarkb | jeblair: or even if rax was mixed up, the uuid for ord wouldn't be valid in iad | 15:12 |
| fungi | clarkb: perhaps. though we did at least confirm that the requested image uuid (according to the debug log) existed in the provider+region corresponding to the request | 15:12 |
| jeblair | stevemar: http://docs.openstack.org/infra/manual/creators.html | 15:12 |
| stevemar | oh the LP stuff... gotta do that | 15:12 |
| *** mtanino has joined #openstack-infra | 15:12 | |
| *** salv-orlando has joined #openstack-infra | 15:13 | |
| fungi | clarkb: but it's still possible the log is lying to us about what it requested where | 15:13 |
| jeblair | clarkb: yeah, but the uuid in hpcloud-b1 _would_ be valide for hpcloud-b2, so i think we actually need to have the providers crossing clouds to explain the hp errors | 15:13 |
| stevemar | jeblair, we have the code here: https://github.com/stevemart/keystone-mapper but i'll go through the steps in the link | 15:13 |
| clarkb | fungi: right so uuid for ord and region is ord, but if the request was actually sent to iad then we could get the error | 15:13 |
| clarkb | jeblair: good point | 15:13 |
| stevemar | created with the cookiecutter | 15:13 |
| jeblair | stevemar: yeah, there should be instructions in there for how to import existing code | 15:13 |
| greghaynes | oh more nodepool asplosions :( | 15:13 |
| *** witek has quit IRC | 15:14 | |
| stevemar | cool cool, marekd did you get the link? http://docs.openstack.org/infra/manual/creators.html | 15:14 |
| *** hemnafk is now known as hemna | 15:14 | |
| fungi | but yeah, if this is something i can test with the nodepool cli, then i think i can pinpoint for sure the patch that introduces this issue | 15:14 |
| jeblair | fungi: ++ | 15:14 |
| fungi | without a bunch of nodepoold restarts | 15:14 |
| marekd | stevemar: yea, thanks. | 15:14 |
| fungi | hopefully we're not looking at the intersection of 2+ issues | 15:15 |
| *** yfried has quit IRC | 15:15 | |
| *** ParsectiX has quit IRC | 15:16 | |
| *** dims_ has joined #openstack-infra | 15:16 | |
| clarkb | I think I see the bug | 15:17 |
| clarkb | you will like this | 15:17 |
| clarkb | patch as soon as I can find all cases | 15:17 |
| clarkb | but it does appear to be a bug in mordreds config reloading change. look at region_name vs region-name | 15:17 |
| fungi | confirmed i can definitely recreate this behavior from a virtualenv, so now to bisect | 15:18 |
| clarkb | fungi: well I think I figured it out but bisecting for independent confirmation is ++ | 15:18 |
| *** dannywilson has joined #openstack-infra | 15:18 | |
| fungi | clarkb: do you think it came in with "Use shade to get the client objects" | 15:19 |
| fungi | if so i'll try the commit right before that was merged | 15:19 |
| greghaynes | fungi: the reloading one is Ie4cf44adefda637b5987dd9d39ec398786086720 | 15:19 |
| jeblair | fungi: try "Consume os-client-config directly" | 15:19 |
| jeblair | fungi: i believe that's the change with the typo clarkb suspects | 15:19 |
| clarkb | yup | 15:20 |
| jeblair | fungi: so "Debug dying task managers" would be the one before that which should work | 15:20 |
| *** mwhagedorn has quit IRC | 15:21 | |
| *** panda has quit IRC | 15:21 | |
| jeblair | clarkb: every kw in that block has the same error | 15:21 |
| greghaynes | clarkb: so we were passing region_name in to shade previously | 15:21 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Check proper providers keys when reading config https://review.openstack.org/186791 | 15:21 |
| timrc | pabelanger, So grafana 2 is mega dumb in some ways... We might have to also customize /usr/share/grafana/public/dashboards/home.json so we get a home dashboard that makes sense for upstream (and in my case currently within HP) | 15:21 |
| clarkb | that should be the fix | 15:21 |
| fungi | jeblair: clarkb: hrm, resetting to that one and updating my venv, i still see dfw aliens in iad | 15:21 |
| *** sputnik13 has joined #openstack-infra | 15:21 | |
| *** panda has joined #openstack-infra | 15:22 | |
| timrc | pabelanger, There's no concept to easily organize things by teams. Using multiple orgs does not work since only one org can be configured to be viewable anonymously. | 15:22 |
| timrc | So you have to get clever with tags and searching. | 15:22 |
| *** salv-orlando has quit IRC | 15:22 | |
| *** dannywilson has quit IRC | 15:23 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add experimental Rally jobs for Manila https://review.openstack.org/186702 | 15:23 |
| jeblair | fungi: we should consider the possibility of two errors... can you go further back looking for the aliens problem? | 15:24 |
| fungi | yeah, seems like there's something missing in my testing methodology. i reset to a ref from several weeks ago and still have dfw aliens in iad | 15:24 |
| fungi | sudo -H -u nodepool ./foo/bin/nodepool alien-list rax-iad 2>/dev/null|grep -c dfw | 15:24 |
| fungi | 119 | 15:24 |
| greghaynes | clarkb: how did you determine that was the issue? AFAICT shade expects region_name | 15:24 |
| *** fijal has quit IRC | 15:25 | |
| *** dannywilson has joined #openstack-infra | 15:25 | |
| jeblair | greghaynes: shade will get region_name | 15:25 |
| clarkb | greghaynes: because we saw that clients had the wrong config so I read the change and realized that it was wrong | 15:25 |
| clarkb | greghaynes: in nodepool.yaml its region-name which we convert to region_name for shade | 15:25 |
| jeblair | greghaynes: the problem is "if 'region_name' in provider: provider['region-name']" | 15:25 |
| clarkb | except ^ tested the wrong name from the yaml | 15:25 |
| greghaynes | oh! | 15:25 |
| greghaynes | derp | 15:26 |
| *** sdake_ has joined #openstack-infra | 15:26 | |
| *** Ala has quit IRC | 15:26 | |
| greghaynes | haha, yea, it would help if we actually checked the config properly | 15:26 |
| jeblair | fungi: hrm. i wonder if somehow your venv-local bin is picking up global stuff | 15:26 |
| clarkb | ya the integratio nstuff I am working on with devstak would probably help that a lot | 15:27 |
| *** fijal has joined #openstack-infra | 15:27 | |
| clarkb | instead of having mocked out providers that just work | 15:27 |
| fungi | i'm testing a ref from march and still seeing this, so seems likely | 15:27 |
| jeblair | shall i install clarkb's patch into prod? | 15:27 |
| fungi | worth a shot | 15:28 |
| clarkb | jeblair: +1 from me | 15:28 |
| jeblair | lemme fix up mordred's too and i'll stack them both | 15:28 |
| fungi | jeblair: interestingly if i reset to a nodepool version prior to the most recent tag then --version does report what i would expect for the source i'm running | 15:28 |
| *** mpavone has quit IRC | 15:29 | |
| jeblair | oh, he fixed it already actually | 15:29 |
| fungi | so it's at least importing some of nodepool from within the venv anyway | 15:29 |
| *** freerunner has quit IRC | 15:29 | |
| fungi | or else --version would report the globally installed one | 15:30 |
| *** sdake has quit IRC | 15:30 | |
| *** annegentle has joined #openstack-infra | 15:30 | |
| *** fawadkhaliq has quit IRC | 15:30 | |
| fungi | also entirely possible the alien-list issue is a very long standing problem | 15:30 |
| greghaynes | clarkb: hrm, I wonder why the reload test didnt catch that | 15:31 |
| greghaynes | clarkb: oh, because its not using nodepool.yaml maybe | 15:31 |
| clarkb | greghaynes: becase the code "works" it just doesn't do what you want | 15:31 |
| *** fijal has quit IRC | 15:31 | |
| greghaynes | yea, but we test that the clients get the correct config in that test | 15:31 |
| clarkb | it just doesn't set api-timeout and region_name | 15:31 |
| fungi | since this was a known problem with hocloud aliens already, i just always treated rax the same because it was easier to pretent the problem was everywhere. so as far as i know it could have been this way for quite a long time | 15:31 |
| greghaynes | clarkb: ah, were not tetsing those config properties either | 15:32 |
| jeblair | nodepool did not stop gracefully, possibly due to a running dib | 15:33 |
| fungi | clarkb: on your fix... wasn't there a recent patch to os-client-config to s/-/_/ on all the keys? i guess that hasn't merged/released yet? | 15:33 |
| clarkb | fungi: thats not really related, in this case we never pass the keys to os-c-c because the checks fail | 15:33 |
| fungi | aha | 15:34 |
| *** amitgandhinz has joined #openstack-infra | 15:34 | |
| greghaynes | yea, its a nodepool issue | 15:34 |
| jeblair | nodepool is in 'delete all the things' phase | 15:35 |
| fungi | oh! got it. this is "nodepool.yaml uses - but clouds.yaml read by shade indexes as _ instead" | 15:35 |
| *** fijal has joined #openstack-infra | 15:35 | |
| fungi | er, read by oscc | 15:36 |
| pabelanger | timrc, okay. Ya I know the API is pretty new, so not expecting everything to work as we expect it. So, hacking could be expected | 15:37 |
| fungi | so the nodepool patch that introduced this mistakenly used the key names from oscc rather than what we have in our nodepool config | 15:39 |
| clarkb | ya and I think that means we get config done purely by oscc which means we lose ocntrol of regions? | 15:40 |
| clarkb | it may not actually fix everything since hpcloud saw the problem too but its images should be valid across all providers | 15:40 |
| fungi | interestingly enough, that patch *seems* to have solved the misidentified aliens issue as well | 15:40 |
| jeblair | mordred: is there a reason you switched from '-' to '_' when you put that in shade/ossc? | 15:40 |
| *** gyee has joined #openstack-infra | 15:41 | |
| greghaynes | I think it was just a legit typo - there is a - to _ mapping that needs to happen just not when checking the config | 15:41 |
| clarkb | fungi: ya I expected it to | 15:41 |
| fungi | jeblair: i think the commit message for https://review.openstack.org/185967 hints at the decision | 15:41 |
| jeblair | greghaynes: it's a problem of our own making though | 15:41 |
| greghaynes | oh, asking why shade or oscc does _ | 15:41 |
| jeblair | fungi: that was _faaaar_ after mordred made the decision to use '_' | 15:42 |
| *** russellb is now known as rustlebee | 15:42 | |
| greghaynes | yaml? | 15:42 |
| fungi | i.e., it's nice to be able to use these as python variable names | 15:42 |
| * greghaynes needs to afk for a bit, gl | 15:42 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Enable NeutronL3HA for the HA job https://review.openstack.org/185461 | 15:42 |
| *** annegentle has quit IRC | 15:42 | |
| fungi | jeblair: well, that patch is far after the choice was made, but i'm assuming the choice was made so that he could directly use them as kwargs without needing to translate them | 15:42 |
| *** fijal has quit IRC | 15:42 | |
| *** annegentle has joined #openstack-infra | 15:43 | |
| jeblair | fungi: sure. and it may be that we all have different ideas about which one is better. but regardless, i really think it's worthwhile for us to pick one and stick with it. i try very hard to do that within a single program. it makes sense to me that as we split things up into different libraries, we would try to be consistent across them | 15:43 |
| jeblair | to prevent exactly this sort of error | 15:43 |
| *** dizquierdo has quit IRC | 15:43 | |
| clarkb | +1 | 15:43 |
| fungi | absolutely | 15:43 |
| *** dansmith is now known as superdan | 15:44 | |
| *** yamamoto has joined #openstack-infra | 15:45 | |
| jeblair | fungi: i only seed iad nodes in alien-list iad | 15:45 |
| *** ayoung has quit IRC | 15:45 | |
| jeblair | and we seem to have spiked to usage/capacity | 15:45 |
| *** kutija has joined #openstack-infra | 15:45 | |
| fungi | yep, i confirmed clarkb's patch seems to have solved that | 15:46 |
| jeblair | usage=capacity even | 15:46 |
| *** Guest98495 has quit IRC | 15:46 | |
| jeblair | oh, sorry missed that | 15:46 |
| fungi | and yes, having a s/-/_/ as a to_kwargs() method in oscc would have been cleaner to achieve that goal, as compared to switching the keys themselves, i agree | 15:46 |
| *** fijal has joined #openstack-infra | 15:48 | |
| jeblair | fungi: oh, does that patch to ossc implement s/-/_/ for the config file? | 15:48 |
| fungi | jeblair: it transforms teh key names as the config is read in, yeah | 15:48 |
| fungi | so within oscc the interpreted yaml is all _ in keys rather than - | 15:49 |
| clarkb | ya so as a followup we can clean up the nodepool change even further to just pass the nodepool keys through | 15:49 |
| clarkb | but we need an oscc release (not sure if that happened yet) | 15:49 |
| *** dims_ is now known as dimsum__ | 15:49 | |
| jeblair | so you can write 'region-name' in the clouds.yaml config file that ossc reads | 15:49 |
| fungi | and oscc sees it as region_name | 15:50 |
| fungi | yep | 15:50 |
| jeblair | but if you pass in the args to the library without reading it... that's not the case, right? | 15:50 |
| clarkb | jeblair: and in the get_one_cloud() function we used to merge it all together | 15:50 |
| nibalizer | mriedem: i am here now | 15:50 |
| nibalizer | whats up | 15:50 |
| fungi | jeblair: yeah, it seems to just be for the yaml loader | 15:51 |
| clarkb | fungi: oh really? I am wrong about get_one_cloud then | 15:51 |
| fungi | jeblair: clarkb: oh, unless it was already doing that in get_one_cloud() before 185967 | 15:51 |
| fungi | looking now | 15:51 |
| *** zz_dimtruck is now known as dimtruck | 15:52 | |
| fungi | ahh, yep. _fix_args() | 15:52 |
| fungi | so the problem solved in 185967 was that the yaml loader was a separate path for adding keys | 15:53 |
| *** tjones1 has joined #openstack-infra | 15:53 | |
| *** fijal has quit IRC | 15:53 | |
| jeblair | fungi: okay, so with that, all paths into oscc support both '-' and '_' ? | 15:53 |
| fungi | jeblair: looks that way, yes | 15:53 |
| mriedem | nibalizer: ? | 15:54 |
| mriedem | nibalizer: looking for mtreinish? | 15:54 |
| *** asselin_ has quit IRC | 15:54 | |
| fungi | jeblair: it looks like the _fix_args() method was introduced last year and is included in tags as far back as 0.3.0 | 15:55 |
| fungi | so we should be safe to rely on that immediately | 15:55 |
| jeblair | but we need a release with 185967 in order to change our clouds.yaml file | 15:55 |
| jeblair | ? | 15:55 |
| mtreinish | mriedem: I don't think I pinged nibalizer, but who knows :) | 15:55 |
| fungi | jeblair: yep | 15:55 |
| jeblair | cool, i think i have achieved understanding | 15:56 |
| *** davideagnello has joined #openstack-infra | 15:56 | |
| fungi | jeblair: oh, actually we already have that | 15:56 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Nova non-voting bandit job https://review.openstack.org/186806 | 15:57 |
| jeblair | fungi: that change is in the release we're using? | 15:57 |
| fungi | 185967 is included as far back as 1.1.0 and 1.2.0 is the latest tag | 15:57 |
| nibalizer | mriedem: not even, mrmartin is who i"m looking for, apologies | 15:57 |
| *** mattfarina has joined #openstack-infra | 15:57 | |
| fungi | jeblair: 1.1.0 was a couple days ago and 1.2.0 was a day ago | 15:58 |
| fungi | fast-paced release schedule ;) | 15:58 |
| *** jistr has quit IRC | 15:58 | |
| jeblair | clarkb, fungi: in nodepool, why do we set the 'cloud' argument along with all the others... shouldn't it be 'cloud' XOR all the others? | 15:58 |
| *** fijal has joined #openstack-infra | 15:59 | |
| clarkb | jeblair: my understanding is that you can set cloud to inherit from clouds.yaml and set the others to override | 15:59 |
| jeblair | oh, also, someone said something about azs... do we need to swing back around to that? | 15:59 |
| *** samuelBartel has quit IRC | 15:59 | |
| *** bknudson has joined #openstack-infra | 15:59 | |
| *** linkedinyou has quit IRC | 15:59 | |
| clarkb | jeblair: so the perfect example for that is region-name. The rax cloud ahs three regions but from nodepool's perspective we want to differentiate | 15:59 |
| jeblair | clarkb: ok, so we 'cloud=rax'+region_name=ord | 16:00 |
| clarkb | we could also have rax-ord, rax-iad, rax-dfw as separate clouds in clouds.yaml | 16:00 |
| clarkb | jeblair: ya | 16:00 |
| jeblair | got it, makes sense | 16:00 |
| *** devvesa has quit IRC | 16:00 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Cleanup subunit capture in run-tox.sh https://review.openstack.org/186809 | 16:01 |
| *** annegentle has quit IRC | 16:01 | |
| *** akurenyshev has joined #openstack-infra | 16:02 | |
| jeblair | so what's the deal with azs? | 16:02 |
| jeblair | also, shall i ask the hpcloud noc to stand down on this issue? :) | 16:03 |
| mtreinish | fungi, clarkb: ^^^ that should be a pretty simple cleanup | 16:03 |
| mordred | clarkb: what's the tl;dr? | 16:04 |
| clarkb | jeblair: I think we can stand down on hpcloud noc | 16:04 |
| mordred | (on phone) | 16:04 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Cleanup subunit capture in run-tox.sh https://review.openstack.org/186809 | 16:04 |
| *** fijal has quit IRC | 16:04 | |
| openstackgerrit | Kyle Mestery proposed openstack/requirements: Add networking-odl to tracked projects list https://review.openstack.org/186812 | 16:04 |
| clarkb | mordred: the check for config updates change was checking for region_name in provider: not region-name | 16:04 |
| clarkb | mordred: so we didn't set region name properly | 16:04 |
| clarkb | jeblair: I am not sure what the deal with asz is | 16:04 |
| clarkb | mordred: which I think led to nodepool sending ord requests to iad and so on | 16:05 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Move networking-odl into the openstack namespace https://review.openstack.org/186813 | 16:05 |
| mordred | ahhhhhh | 16:05 |
| mordred | wow | 16:05 |
| mordred | that makes sense | 16:05 |
| clarkb | so we got a proper response from nova | 16:05 |
| *** jistr has joined #openstack-infra | 16:06 | |
| mordred | yah ... definitely need to take a better pass at the - _ thing. it's obvs to brittle as is | 16:07 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Initialize new keystone-mapper project https://review.openstack.org/186817 | 16:07 |
| jeblair | mordred: i _think_ '-' is safe to use everywhere, we got bit because we were doing a translation | 16:07 |
| *** tjones1 has quit IRC | 16:08 | |
| *** krtaylor has quit IRC | 16:08 | |
| *** jlanoux has quit IRC | 16:08 | |
| *** fijal has joined #openstack-infra | 16:08 | |
| *** akurenyshev has left #openstack-infra | 16:08 | |
| jeblair | i'm going to try simplifying this in nodepool... | 16:08 |
| mordred | jeblair: I think scrollback says this ... but _ in occ is due to the passing to kwargs. I like fungi's suggestion of a to_kwargs method | 16:08 |
| *** vaidy_ has quit IRC | 16:08 | |
| mordred | jeblair: ++ | 16:08 |
| *** jistr has quit IRC | 16:09 | |
| *** tteggel has quit IRC | 16:09 | |
| jeblair | mordred: we've convinced ourselves that ossc accepts '-' in all the places we interface with it | 16:09 |
| mordred | it should | 16:09 |
| mordred | if it doesn't, it's a bug. we even have some tests | 16:10 |
| *** puranamr has joined #openstack-infra | 16:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Simplify oscc argument conversion https://review.openstack.org/186819 | 16:10 |
| fungi | yep, the only concern with to_kwargs() is i think if we're using oscc to load a config and then want to reference things in its dict from outside | 16:10 |
| jeblair | there ^ | 16:10 |
| mordred | fungi good point | 16:10 |
| *** vaidy_ has joined #openstack-infra | 16:10 | |
| fungi | er, only concern to which i think to_kwargs() is a potential solution i mean | 16:10 |
| fungi | but for stuffing keys into oscc from outside, it's not an issue | 16:10 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add stable branch ACLs for the neutron *aas repos https://review.openstack.org/186820 | 16:10 |
| *** freerunner has joined #openstack-infra | 16:11 | |
| *** tteggel has joined #openstack-infra | 16:11 | |
| jeblair | clarkb, mordred: the 'rackspace' cloud in clouds.yaml has a list for region-name.... what does that mean? | 16:12 |
| jeblair | (actually, so does hpcloud) | 16:12 |
| mordred | jeblair: fwiw, we can also pass project-id ... it's a synonym in occ | 16:12 |
| jeblair | mordred: cool, will simplify further :) | 16:12 |
| *** dtantsur is now known as dtantsur|afk | 16:12 | |
| *** fijal has quit IRC | 16:13 | |
| *** Sukhdev has quit IRC | 16:13 | |
| *** tjones1 has joined #openstack-infra | 16:13 | |
| fungi | when did we start gating nodepool on a coverage job? | 16:13 |
| mordred | jeblair: it's shorthand that means you don't have to make a rax-dfw and a rax-iad cloud. you give a list then get-all-clouds gives you a cloud for each cloud region | 16:13 |
| clarkb | fungi: when we decided we needed much better testing I added that to help us track our progress | 16:13 |
| fungi | clarkb: got it | 16:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Simplify oscc argument conversion https://review.openstack.org/186819 | 16:14 |
| clarkb | fungi: not sure why it failed there | 16:14 |
| fungi | interestingly it looks like it failed on a unit test in the coverage job | 16:14 |
| *** rwsu has joined #openstack-infra | 16:15 | |
| fungi | OSError: [Errno 2] No such file or directory: 'clouds.yaml' | 16:15 |
| *** mwhagedorn has joined #openstack-infra | 16:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Switch to using '-' in clouds.yaml https://review.openstack.org/186823 | 16:15 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Cleanup subunit capture in run-tox.sh https://review.openstack.org/186809 | 16:15 |
| greghaynes | fungi huh - Is that the cleanup method in test_shade_integration? | 16:15 |
| *** alexsyip has joined #openstack-infra | 16:15 | |
| clarkb | greghaynes: yes | 16:16 |
| fungi | greghaynes: yep | 16:16 |
| greghaynes | I didnt guard it because I shouldnt have to, but it does do an rm clouds.yaml | 16:16 |
| *** aduarte has joined #openstack-infra | 16:16 | |
| *** fijal has joined #openstack-infra | 16:17 | |
| fungi | maybe racy | 16:17 |
| greghaynes | ah, yes | 16:17 |
| greghaynes | thats it | 16:17 |
| greghaynes | it diesnt tmpdir | 16:17 |
| *** vaidy_ has quit IRC | 16:17 | |
| *** unicell has quit IRC | 16:17 | |
| greghaynes | it needs to tmpdir and cd into it instead | 16:17 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Initialize new keystone-mapper project https://review.openstack.org/186817 | 16:18 |
| * greghaynes is on phone irc atm so cant fix | 16:18 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 16:18 | |
| *** vahidh has joined #openstack-infra | 16:18 | |
| *** Sukhdev has joined #openstack-infra | 16:20 | |
| *** fijal has quit IRC | 16:21 | |
| *** markus_z has quit IRC | 16:22 | |
| *** ashleighfarnham has joined #openstack-infra | 16:22 | |
| *** fijal has joined #openstack-infra | 16:22 | |
| *** koolhead17 has joined #openstack-infra | 16:23 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Initialize new keystone-mapper project https://review.openstack.org/186817 | 16:23 |
| *** gokrokve has quit IRC | 16:23 | |
| *** amotoki has quit IRC | 16:24 | |
| *** spzala has quit IRC | 16:24 | |
| *** spzala has joined #openstack-infra | 16:25 | |
| pleia2 | good morning | 16:25 |
| nibalizer | o/ | 16:25 |
| clarkb | jeblair: I think 186823 can be a revert too since I had to convert to _ prior to oscc being updated | 16:25 |
| clarkb | jeblair: and comment on the nodepool config simplification change | 16:27 |
| jeblair | clarkb: i'm feeling lazy, is it important we make 186823 a revert? :) | 16:27 |
| *** mmedvede has quit IRC | 16:27 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 16:28 | |
| clarkb | jeblair: nope, just wanted to note that | 16:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Simplify oscc argument conversion https://review.openstack.org/186819 | 16:28 |
| jeblair | thx | 16:28 |
| zaro | morning | 16:29 |
| *** nfedotov has quit IRC | 16:29 | |
| *** sarob has joined #openstack-infra | 16:29 | |
| *** gokrokve has joined #openstack-infra | 16:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/186829 | 16:30 |
| zaro | electrical: regarding https://review.openstack.org/#/c/131404/ thanks for checking the params. I looks ok but might be better to use one of the other exceptions in errors.py? | 16:31 |
| *** shashankhegde has joined #openstack-infra | 16:32 | |
| *** freerunner has quit IRC | 16:32 | |
| electrical | zaro: might make more sense yeah. Will make a change for that in the weekend or Monday. | 16:33 |
| *** mmedvede has joined #openstack-infra | 16:34 | |
| *** Swami has joined #openstack-infra | 16:35 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/nodepool: Adding a cleanup timeout to nodepool https://review.openstack.org/150955 | 16:37 |
| *** mwhagedorn has quit IRC | 16:37 | |
| *** hashar has quit IRC | 16:38 | |
| *** esker has quit IRC | 16:38 | |
| *** jerryz has joined #openstack-infra | 16:41 | |
| *** abregman has joined #openstack-infra | 16:41 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 16:43 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements https://review.openstack.org/186834 | 16:44 |
| *** ociuhandu has joined #openstack-infra | 16:44 | |
| zaro | electrical: sorry i didn't point that out in last review. | 16:44 |
| *** mpaolino has quit IRC | 16:46 | |
| *** unicell1 has joined #openstack-infra | 16:46 | |
| *** gokrokve has quit IRC | 16:46 | |
| *** derekh has quit IRC | 16:48 | |
| *** freerunner has joined #openstack-infra | 16:48 | |
| *** amitgandhinz has quit IRC | 16:48 | |
| *** amitgandhinz has joined #openstack-infra | 16:49 | |
| zaro | waynr: saw you mentioned that we should not make further changes to jjb core until the big refactor. would you like to discuss? | 16:50 |
| electrical | zaro: no worries. | 16:51 |
| *** weshay has quit IRC | 16:53 | |
| *** weshay has joined #openstack-infra | 16:53 | |
| zaro | electrical, zxiiro: any opinion on which you prefer? https://review.openstack.org/178836 OR https://review.openstack.org/186176 | 16:53 |
| adam_g | hi, what caches git repos these days on devstack slave nodes? trying to get a stackforge project tested as a devstack plugin but failing because of RECLONE=False | 16:54 |
| pleia2 | today is a slow meeting day, can a core have a look at https://review.openstack.org/#/c/182368/ so I can +A it after the 1800 meeting this afternoon? | 16:54 |
| *** spzala has quit IRC | 16:55 | |
| clarkb | adam_g: you need to add your project to the devstack-gate PROJECTS var | 16:55 |
| clarkb | adam_g: the glusterfs devstack plugin job is probably a decent place to look at | 16:55 |
| *** spzala has joined #openstack-infra | 16:56 | |
| *** patrickeast has joined #openstack-infra | 16:56 | |
| electrical | zaro: I think waynr meant no changes to yaml parsing. Plugins them selves should still be good as that part won't change. | 16:56 |
| openstackgerrit | sean mooney proposed openstack-infra/project-config: Add intel-nfv-ci-tests repo to stackfoge https://review.openstack.org/186839 | 16:56 |
| *** gokrokve has joined #openstack-infra | 16:57 | |
| *** hdd has quit IRC | 16:57 | |
| *** jogo is now known as flashgordon | 16:57 | |
| adam_g | clarkb, you mean on-the-fly in the shell builder? i thoguth i was, maybe not. ive been using ec2-api as a reference | 16:57 |
| *** koolhead17 has quit IRC | 16:57 | |
| *** gokrokve has quit IRC | 16:57 | |
| *** mdrnstm has joined #openstack-infra | 16:58 | |
| *** tjones1 has quit IRC | 16:58 | |
| *** gokrokve has joined #openstack-infra | 16:58 | |
| *** mdrnstm is now known as Guest49465 | 16:58 | |
| *** Guest49465 is now known as morgan | 16:58 | |
| *** morgan has quit IRC | 16:58 | |
| *** morgan has joined #openstack-infra | 16:58 | |
| *** dguerri is now known as dguerri`away | 16:58 | |
| *** mpaolino has joined #openstack-infra | 16:58 | |
| *** morgan is now known as mdrnstm | 16:59 | |
| *** fawadkhaliq has joined #openstack-infra | 16:59 | |
| flashgordon | clarkb: you see jesse Pretorius's respons to the ELK thread | 16:59 |
| clarkb | adam_g: yes on the fly there | 17:00 |
| adam_g | cool | 17:00 |
| *** armax has quit IRC | 17:00 | |
| mordred | fungi, jeblair: y'all saw that i disabled puppet on nodepool, yeah? | 17:00 |
| clarkb | adam_g: that tells d-g to sync it properly for you using the cache and pulling any new updates | 17:00 |
| *** jsavak has quit IRC | 17:00 | |
| zaro | electrical: i guess that's what i meant by jjb core :) | 17:00 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Add relevant projects to $PROJECTS for akanda jobs https://review.openstack.org/186840 | 17:01 |
| clarkb | flashgordon: ya currently chatting in #openstack | 17:02 |
| *** xyang1 has quit IRC | 17:02 | |
| *** jsavak has joined #openstack-infra | 17:02 | |
| *** timcline has quit IRC | 17:03 | |
| openstackgerrit | sean mooney proposed openstack-infra/project-config: Add intel-nfv-ci-tests repo to stackfroge https://review.openstack.org/186839 | 17:04 |
| *** salv-orlando has joined #openstack-infra | 17:04 | |
| *** mwhagedorn has joined #openstack-infra | 17:05 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Allow inclusion of multiple files https://review.openstack.org/171591 | 17:05 |
| flashgordon | clarkb: awesome | 17:06 |
| *** jsavak has quit IRC | 17:07 | |
| *** jsavak has joined #openstack-infra | 17:07 | |
| *** otter768 has joined #openstack-infra | 17:08 | |
| *** dannywilson has quit IRC | 17:08 | |
| *** sdake has joined #openstack-infra | 17:09 | |
| *** dannywilson has joined #openstack-infra | 17:09 | |
| *** achanda has quit IRC | 17:10 | |
| *** achanda_ has joined #openstack-infra | 17:10 | |
| *** salv-orlando has quit IRC | 17:10 | |
| *** ildikov has quit IRC | 17:10 | |
| *** patrickeast has quit IRC | 17:11 | |
| *** patrickeast has joined #openstack-infra | 17:11 | |
| *** sdake_ has quit IRC | 17:12 | |
| zxiiro | zaro: both implementations appear to be exactly the same just in different orders | 17:12 |
| *** otter768 has quit IRC | 17:12 | |
| *** bknudson has quit IRC | 17:12 | |
| zxiiro | zaro: i'd probably go with 178836 since I feel it does more thorough testing though | 17:13 |
| zxiiro | zaro: although 186176 feels a little cleaner in my opinoin | 17:14 |
| *** krtaylor has joined #openstack-infra | 17:17 | |
| electrical | zaro: haha sorry yeah. Forgot he mentioned that. Was on the London tube so had bad Internet. | 17:17 |
| *** SumitNaiksatam has quit IRC | 17:19 | |
| *** sdake_ has joined #openstack-infra | 17:20 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:20 | |
| jeblair | mordred: i did not; does it need to stay disabled? | 17:21 |
| timrc | Is there a convention for exporting a variable in a gate check that a post job can then use? | 17:22 |
| *** tjones1 has joined #openstack-infra | 17:22 | |
| *** hdd has joined #openstack-infra | 17:23 | |
| *** sdake has quit IRC | 17:23 | |
| anteaya | dhellmann SpamapS I can't envison what jobs you are thinking about that need to be mapped to owners, perhaps I could request a few examples? | 17:23 |
| fungi | jeblair: i think it should be safe to reenable as long as we get the fix merged before the next time we restart nodepool. i'll go ahead and enable it | 17:23 |
| dhellmann | anteaya: the functional testing job for zmq for oslo.messaging would be one example, since we have a sub-team for zmq support | 17:24 |
| *** koolhead17 has joined #openstack-infra | 17:24 | |
| fungi | jeblair: i believe the main reason it was disabled was that mordred also lowered the max api call rate for hpcloud because we were exceeding our quota for it, but should be long enough now that it'll play nice again | 17:25 |
| jeblair | fungi: oh, right. makes sense. ++ | 17:25 |
| *** vponomaryov has quit IRC | 17:25 | |
| *** cody-somerville has quit IRC | 17:25 | |
| *** sdake has joined #openstack-infra | 17:26 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 17:26 | |
| fungi | yep, that's the only outstanding config divergence, just checked | 17:26 |
| openstackgerrit | Merged openstack-infra/nodepool: Check proper providers keys when reading config https://review.openstack.org/186791 | 17:28 |
| mordred | yah. that and I a applied the debug patch by hand | 17:28 |
| anteaya | dhellmann: where are the oslo subteams listed? | 17:29 |
| fungi | zuul's pretty steadily chipping away at its backlog, so i think we're in good shape since the fix went in | 17:29 |
| anteaya | dhellmann: I can find the liasons page and you used to have a subteams page | 17:29 |
| anteaya | didn't you? | 17:29 |
| dhellmann | anteaya: well, that's sort of the point of starting this discussion -- we don't have a place to list them, yet, and we're trying to figure out a good place | 17:29 |
| dhellmann | I don't think we did have anything formal | 17:29 |
| *** sdake_ has quit IRC | 17:29 | |
| anteaya | okay are there jobs other than olso jobs that fit SpamapS' criteria? | 17:30 |
| dhellmann | we could just put them in the wiki, but I think SpamapS's point that it would be useful to generalize this somehow is worth thinking about | 17:30 |
| anteaya | my concern is keeping the information up to date | 17:30 |
| dhellmann | the postgresql and pypy jobs come to mind | 17:30 |
| anteaya | for oslo? | 17:31 |
| dhellmann | sure | 17:31 |
| dhellmann | anyone can have their unit tests run under pypy | 17:31 |
| dhellmann | and the postgresql job is one of the dsvm jobs | 17:31 |
| fungi | that's why i suggested trying to store contact metadata in the job instantiations and then transform those into some sort of reference publication | 17:31 |
| *** SumitNaiksatam has quit IRC | 17:31 | |
| dhellmann | fungi: yeah, that makes some sense, although it's not clear what level we would need to do that with all of the templating we have | 17:32 |
| jeblair | we could start by just putting them in comments in jjb | 17:32 |
| jeblair | or in zuul | 17:32 |
| fungi | jjb comments assume that any job built from a particular template would have the same contact. not sure if that would necessarily always be the case but maybe it's close enough | 17:32 |
| anteaya | I think both fungi's and jeblair's suggestions have a higher chance of maintaining relevant data | 17:32 |
| *** achanda_ has quit IRC | 17:32 | |
| jeblair | or.... | 17:32 |
| dhellmann | comments might be a good start | 17:33 |
| *** SumitNaiksatam has joined #openstack-infra | 17:33 | |
| jeblair | the _way_ we could put the metadata in jjb could be by invoking a shell builder that echoes the information, that way it ends up directly in the job console output | 17:33 |
| *** achanda has joined #openstack-infra | 17:33 | |
| * dhellmann tries to remember enough about jjb to envision that | 17:33 | |
| fungi | also i'm not sure about scalability of tracking contacts for _all_ jobs since we're rapidly approaching 6000 already | 17:34 |
| *** timcline has joined #openstack-infra | 17:34 | |
| jeblair | fungi: but only a few classes. honestly, we don't need to know the contact for the py27 nova job | 17:34 |
| dhellmann | yeah, this may really be most important for the "odd" jobs where some specialist is needed | 17:34 |
| jeblair | we just need to know the contact for the bobSQL job | 17:34 |
| *** tjones1 has quit IRC | 17:34 | |
| *** unicell has joined #openstack-infra | 17:34 | |
| jeblair | (hint, it's not bob) | 17:34 |
| fungi | bobsql rules! | 17:34 |
| mordred | bobSQL is the best | 17:34 |
| mordred | jeblair: is it Alice? | 17:35 |
| *** mrmartin has joined #openstack-infra | 17:35 | |
| fungi | this is quickly going down a rabbit hole | 17:35 |
| *** tjones1 has joined #openstack-infra | 17:35 | |
| dhellmann | fungi: the rabbit jobs need an owner, too ;-) | 17:35 |
| fungi | bwahahaha | 17:35 |
| mordred | you should replace rabbit with imap | 17:35 |
| dhellmann | mordred: when you write it and have a ci job, we can talk about it | 17:36 |
| anteaya | dhellmann: the macros might help you envision jeblair's suggestion: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml | 17:36 |
| * dhellmann goes back to the ever growing list of libraries to be released on monday | 17:36 | |
| *** unicell1 has quit IRC | 17:36 | |
| *** HeOS has quit IRC | 17:36 | |
| * anteaya looks forward to dhellmann's monday email stack | 17:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add a contact macro https://review.openstack.org/186852 | 17:37 |
| jeblair | anteaya, dhellmann, fungi: ^ quick sketch | 17:37 |
| dhellmann | jeblair: I like it. Maybe add irc, too. | 17:37 |
| jeblair | ++ | 17:38 |
| *** bdemers has joined #openstack-infra | 17:38 | |
| *** timcline has quit IRC | 17:38 | |
| jeblair | it's worth pushing that into a couple of edge cases to... i'm pretty sure you can also templatize that, so for, say, the devstack plugin jobs that share a template, you could actually require the contact information to instantiate the template | 17:39 |
| jeblair | i'll sketch that too real quick | 17:39 |
| fungi | looks like a good compromise. and i guess we can pass in a contact parameter from template parameters instantiated in projects.yaml if a given job-template needs different owners for different cases | 17:39 |
| dhellmann | nice, I like it | 17:39 |
| fungi | i think i just said what you said using different words | 17:39 |
| fungi | but yes, great idea | 17:39 |
| jeblair | fungi: yeah, i think it's twice as good now | 17:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add a contact macro https://review.openstack.org/186852 | 17:42 |
| jeblair | with irc and an extra layer of templating for devstack plugins | 17:43 |
| dimsum__ | dhellmann: should i take the week off? :) | 17:44 |
| dhellmann | dimsum__: most of these aren't actually oslo libs :-) | 17:44 |
| *** mpaolino has quit IRC | 17:44 | |
| dimsum__ | dhellmann: kidding of course :) yes, i realized | 17:44 |
| *** annegentle has joined #openstack-infra | 17:45 | |
| *** sridhar_ram has joined #openstack-infra | 17:45 | |
| dhellmann | dimsum__: but I may take the rest of the week off after lunch monday | 17:46 |
| *** ivar-lazzaro has joined #openstack-infra | 17:46 | |
| * dhellmann drops libs and walks away | 17:46 | |
| dimsum__ | haha | 17:46 |
| jeblair | dhellmann: i'm assuming you have library releases on some kind of deadman switch :) | 17:46 |
| jeblair | if anything happens to doug, the libraries get automatically released! | 17:47 |
| dhellmann | jeblair: fantastic idea | 17:47 |
| dhellmann | #jobsecurity | 17:47 |
| anteaya | ha ha ha | 17:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add a tool to update gerrit groups https://review.openstack.org/186596 | 17:49 |
| *** xyang1 has joined #openstack-infra | 17:50 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Move networking-odl into the openstack namespace https://review.openstack.org/186813 | 17:50 |
| *** timcline has joined #openstack-infra | 17:51 | |
| anteaya | jeblair: I posted the link to your patch in a reply to SpamapS' thread on the ml | 17:52 |
| openstackgerrit | Terry Howe proposed openstack-infra/project-config: Remove OSC stable gate from cliff https://review.openstack.org/186856 | 17:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add a tool to update gerrit groups https://review.openstack.org/186596 | 17:52 |
| jeblair | anteaya: thanks | 17:53 |
| *** teran has quit IRC | 17:56 | |
| anteaya | welcome | 17:56 |
| *** teran has joined #openstack-infra | 17:56 | |
| *** ihrachyshka has quit IRC | 17:57 | |
| *** mrmartin has quit IRC | 17:59 | |
| *** teran has quit IRC | 18:00 | |
| *** achanda_ has joined #openstack-infra | 18:00 | |
| *** jsavak has quit IRC | 18:02 | |
| *** achanda has quit IRC | 18:02 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Nova non-voting bandit job https://review.openstack.org/186806 | 18:02 |
| *** jsavak has joined #openstack-infra | 18:02 | |
| *** spzala has quit IRC | 18:03 | |
| *** sridhar_ram1 has joined #openstack-infra | 18:03 | |
| *** spzala has joined #openstack-infra | 18:03 | |
| *** sridhar_ram has quit IRC | 18:04 | |
| *** jsavak has quit IRC | 18:04 | |
| *** jsavak has joined #openstack-infra | 18:05 | |
| *** SumitNaiksatam has quit IRC | 18:05 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:06 | |
| *** ashleighfarnham has quit IRC | 18:06 | |
| *** bknudson has joined #openstack-infra | 18:07 | |
| *** tjones1 has quit IRC | 18:07 | |
| *** sdake_ has joined #openstack-infra | 18:07 | |
| *** annegentle has quit IRC | 18:08 | |
| *** annegentle has joined #openstack-infra | 18:09 | |
| jeblair | https://review.openstack.org/#/admin/groups/925,members | 18:09 |
| *** tjones1 has joined #openstack-infra | 18:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add a script to configure the infra group structure https://review.openstack.org/186861 | 18:11 |
| jeblair | that script created that group ^ | 18:11 |
| waynr | zaro: yo, i am not on a reliable connection but i can discuss avoiding jjb core changes for a bit | 18:11 |
| *** bknudson has quit IRC | 18:12 | |
| *** sdake has quit IRC | 18:12 | |
| *** sridhar_ram1 has quit IRC | 18:14 | |
| *** bknudson has joined #openstack-infra | 18:14 | |
| *** fallenpegasus has joined #openstack-infra | 18:14 | |
| waynr | i have two primary concerns, first off i am worried about additional tech debt being added to the yamlparser making the eventual rewrite (which in my opinion is going to be necessary sooner or later) that much more difficult as we will have that many more features to support with the rewrite | 18:14 |
| *** pradk has joined #openstack-infra | 18:15 | |
| *** sridhar_ram has joined #openstack-infra | 18:16 | |
| waynr | second, i want to make sure that some of the breaking changes that have been recently submitted (deep merge of template parameters onto default parameters rather than override and reordering of parameter overrides are two recent examples) make it in on a major version change | 18:16 |
| anteaya | jeblair: so one thing the new infrastructure structure spec didn't deal with was quorum during votes | 18:17 |
| anteaya | jeblair: I don't know how many actual people that gerrit group translates too, but it might be over 30 | 18:18 |
| *** spzala has quit IRC | 18:18 | |
| anteaya | if so, we might have difficulty with quorum | 18:18 |
| jeblair | anteaya: eventually, yes. right nowe it's more like 12. | 18:18 |
| anteaya | ah okay | 18:18 |
| *** spzala has joined #openstack-infra | 18:18 | |
| waynr | additionally to that, i think it would actually be a good idea to put some serious thought into the direction we want to take the behavior of the yamlparser moving forward--up to now it seems that many of the features have been organically "grown" into JJB over time which isn't necessarily a bad thing but i would like to be a little more intentional moving forward | 18:18 |
| anteaya | we seem to be able to gather 7 or so folks for most meetings now | 18:18 |
| jeblair | anteaya: voting will take place over the course of days. in that case, i don't think quorum will be a problem. | 18:19 |
| jeblair | anteaya: voting doesn't happen in meetings. voting happens in gerrit. | 18:19 |
| anteaya | ah okay I had interpreted voting happening during meetings | 18:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Publish git-review documentation https://review.openstack.org/186168 | 18:19 |
| lifeless | mattt: hi? | 18:19 |
| jeblair | http://logs.openstack.org/52/186852/2/check/gate-project-config-layout/4b41e7e/console.html | 18:19 |
| jeblair | that is a very unhelpful output from jjb | 18:20 |
| *** Sukhdev has quit IRC | 18:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add a contact macro https://review.openstack.org/186852 | 18:21 |
| fungi | anteaya: i think the expectation is that council voting on infra specs will happen much in the same way that tc voting happens on resolutions | 18:21 |
| *** Sukhdev has joined #openstack-infra | 18:21 | |
| anteaya | fungi: ah okay | 18:21 |
| jeblair | except not actually in meetings :) | 18:22 |
| fungi | well, yeah. the way the tc is sort of expected to vote but seems often not to get around to until prodded in the meeting | 18:22 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Add a usage message for list_unreleased_changes.sh https://review.openstack.org/186867 | 18:22 |
| waynr | zaro: stepping away from laptop for a bit, would like to continue discussing throughout the day | 18:22 |
| jeblair | i think the infra meeting will basically be: "vote on this by <some>day"; next topic! | 18:22 |
| *** rockyg has quit IRC | 18:22 | |
| fungi | i think that's a great idea | 18:23 |
| *** fallenpegasus has quit IRC | 18:23 | |
| jeblair | so it's a coordination/communication point, but not a bottleneck. i'm thinking 24-48 hours from meeting till close of vote. | 18:23 |
| *** arxcruz has quit IRC | 18:24 | |
| electrical | waynr: would love to give my input regarding rewrite. | 18:24 |
| *** gokrokve_ has joined #openstack-infra | 18:24 | |
| *** marun has quit IRC | 18:24 | |
| *** crc32 has joined #openstack-infra | 18:24 | |
| jeblair | in fact, it's almost the opposite of the tc -- the tc wants to end voting during a meeting, and i want to start it during a meeting :) | 18:24 |
| electrical | When designing happens :) | 18:24 |
| *** fedexo has joined #openstack-infra | 18:24 | |
| waynr | electrical: excellent! | 18:25 |
| *** armax has joined #openstack-infra | 18:25 | |
| waynr | electrical: i want to wait until we stabilize the JJB API before addressing the yamlparser rewrite | 18:25 |
| waynr | just FYI | 18:25 |
| *** fallenpegasus has joined #openstack-infra | 18:26 | |
| *** alexpilotti has joined #openstack-infra | 18:26 | |
| *** zz_zz_zz_ja is now known as zz_zz_zz_zz_ja | 18:26 | |
| fungi | jeblair: that's fair. we don't need these closed as approved resolutions within the meeting, so we have more flexibility there | 18:27 |
| zaro | electrical: here is your opportunity, https://review.openstack.org/#/c/182542/ | 18:27 |
| *** gokrokve has quit IRC | 18:27 | |
| *** pradk has quit IRC | 18:27 | |
| waynr | well i guess if someone wants to work on a yamlparser spec while we work on the JJB API stabilization that would be good | 18:27 |
| waynr | zaro: i'll be addressing comments on that patch tomorrow night | 18:28 |
| lifeless | whats the current refreshment interval on base images? | 18:28 |
| zaro | waynr: ok, i guess i thought you were referring to your current spec about jjb api. | 18:28 |
| lifeless | [wondering when we get pip 7 everywhere] | 18:28 |
| zaro | waynr: do you feel we should free for the jjb api work? | 18:29 |
| *** gokrokve_ has quit IRC | 18:29 | |
| fungi | lifeless: it's daily unless image updating fails or has to be rolled back to the next oldest due to a bug | 18:29 |
| *** freerunner has quit IRC | 18:29 | |
| waynr | zaro: oh that current spec is purely intended to be about the JJB API, I'll be trying to stay away from behavior changes or actually rewriting internals unless necessary for cleaning up the API | 18:29 |
| lifeless | fungi: ok so we're seeing pip 6.1.1 in https://review.openstack.org/#/c/182423/ | 18:29 |
| *** freerunner has joined #openstack-infra | 18:30 | |
| fungi | lifeless: also as noted previously we pin to a specific virtualenv release, so unless you're upgrading pip within the virtualenv you get whatever version is bundled. at this point i think we have a 12.1.1 virtualenv getting installed | 18:30 |
| *** crc32 has quit IRC | 18:30 | |
| lifeless | fungi: are we currently rolled back, or has it been failing all week ? | 18:30 |
| lifeless | fungi: OH | 18:30 |
| *** crc32 has joined #openstack-infra | 18:30 | |
| fungi | lifeless: we can manually bump virtualenv again. we just did it a couple weeks ago | 18:30 |
| lifeless | fungi: ok, whats the process for updating virtualenv? having latest pip but not latest virtualenv doesn't make much sense to me :) | 18:30 |
| *** pradk has joined #openstack-infra | 18:30 | |
| waynr | zaro: not sure what you mean | 18:30 |
| zaro | waynr: ok, maybe we should hold off on discussing yaml parser thing until there's a spec? | 18:30 |
| fungi | lifeless: it's in a puppet manifest. finding it for you now | 18:30 |
| lifeless | [I mean, I'm entirely fine with pinned - that was my safety argument on requirements, just mechanism] | 18:30 |
| *** jsavak has quit IRC | 18:31 | |
| fungi | lifeless: in openstack-infra/system-config update $desired_virtualenv in modules/openstack_project/manifests/template.pp | 18:31 |
| lifeless | harlowja: dhellmann: dimsum__: FYI ^ | 18:31 |
| *** jsavak has joined #openstack-infra | 18:31 | |
| *** gokrokve has joined #openstack-infra | 18:31 | |
| *** ashleighfarnham has joined #openstack-infra | 18:32 | |
| fungi | lifeless: the next image updates that happen after it merges (normally scheduled to start daily at 14:14 utc) should have the specified version unless image builds fail or get manually rolled back | 18:32 |
| *** freerunner has quit IRC | 18:32 | |
| zaro | waynr: sorry, do you feel we should freeze jjb core for the JJB API work that's outlined in your spec? | 18:32 |
| *** patrickeast has quit IRC | 18:32 | |
| openstackgerrit | lifeless proposed openstack-infra/system-config: Update virtualenv to get pip 7.0.1. https://review.openstack.org/186869 | 18:33 |
| openstackgerrit | Merged openstack-infra/project-config: Update infra-specs voting rules https://review.openstack.org/185792 | 18:33 |
| lifeless | fungi: I'm obviously still confused about system-config vs project-config, I would have anticipated specific pins like that to be project-config. | 18:34 |
| *** marun has joined #openstack-infra | 18:35 | |
| fungi | lifeless: project-config is data files, specifically ones used to drive project-specific things (ci job definitions, project creation/acls, et cetera) | 18:35 |
| waynr | zaro: no, i want to freeze "core" work but only related to yamlparser feature additions, bug fixes should always be allowed | 18:35 |
| *** mwhagedorn has quit IRC | 18:35 | |
| *** crc32 has quit IRC | 18:35 | |
| fungi | lifeless: also our node building data/scripts are there, but no puppet modules | 18:35 |
| lifeless | fungi: yeah - exactly what I thought. so why isn't virtualenv controlled there :) | 18:36 |
| waynr | and the freeze that i want isn't related to the API work in my spec but to the yamlparser rewrite that i am certain will be necessary | 18:36 |
| fungi | lifeless: system-config is a puppet module named openstack_project, plus a handful of utilities | 18:36 |
| lifeless | fungi: not the mechanism, the version | 18:36 |
| waynr | okay, stepping away | 18:36 |
| *** mwhagedorn has joined #openstack-infra | 18:36 | |
| *** jsavak has quit IRC | 18:37 | |
| fungi | lifeless: yeah, we could probably parameterize $desired_virtualenv in that class and then pass it down from nodepool prep scripts/dib elements | 18:37 |
| *** koolhead17 has quit IRC | 18:37 | |
| *** marun has quit IRC | 18:37 | |
| lifeless | anyhoo | 18:37 |
| *** marun has joined #openstack-infra | 18:37 | |
| lifeless | plox +A, can has shiny | 18:37 |
| fungi | lifeless: there's a lot of data in the openstack_project puppet module which could be extracted out and turned into more class params, just nobody has gotten to it yet | 18:38 |
| lifeless | do we reuse unittest nodes? | 18:39 |
| fungi | we do not | 18:39 |
| *** jsavak has joined #openstack-infra | 18:39 | |
| fungi | as for parameterization, there was a set of changes a group at mirantis proposed to turn every string in all of our puppet modules into a new class parameter, but that was overkill. some of the gozer admins have been selectively parameterizing some bits and that's going a bit better | 18:40 |
| *** crc32 has joined #openstack-infra | 18:40 | |
| *** annegentle has quit IRC | 18:41 | |
| *** patrickeast has joined #openstack-infra | 18:42 | |
| *** marun has quit IRC | 18:42 | |
| openstackgerrit | Sahdev Zala proposed openstack-infra/project-config: Add #openstack-heat-translator to the list of irc channels https://review.openstack.org/186871 | 18:42 |
| *** marun has joined #openstack-infra | 18:43 | |
| nibalizer | fungi: yep | 18:43 |
| *** patrickeast has quit IRC | 18:43 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Include HEAD SHA in unreleased log output https://review.openstack.org/186872 | 18:43 |
| *** mpaolino has joined #openstack-infra | 18:43 | |
| *** mpaolino has quit IRC | 18:43 | |
| nibalizer | also this https://review.openstack.org/#/c/177320/ would enable a lot of the things that are currently code in system-config to become big blobs of yaml | 18:44 |
| nibalizer | we at -infra love our yaml | 18:44 |
| * fungi would marry yaml if polygamy were legal in his jurisdiction | 18:45 | |
| fungi | also restructuredtext | 18:45 |
| *** teran has joined #openstack-infra | 18:46 | |
| anteaya | move to utah | 18:46 |
| anteaya | mountains | 18:47 |
| uberjay | is anyone else seeing a slow rise in the memory usage of zuul-server? mine grew about 100M in the past 6 hours, up to 829M | 18:47 |
| *** mdrnstm has quit IRC | 18:47 | |
| fungi | uberjay: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all looks like it expores some cache daily | 18:48 |
| fungi | er, expires | 18:48 |
| *** mdrnstm has joined #openstack-infra | 18:49 | |
| *** mdrnstm is now known as Guest58148 | 18:49 | |
| uberjay | mm, but that's for the whole system, not just the zuul process. or is there not much else running on there. (if so, uh, maybe i don't have enough memory dedicated to this VM. yikes!) | 18:50 |
| zaro | waynr: ok. makes sense to me. should pick up discussion with pelix when he becomes available. Maybe you can keep an eye on jjb core changes coming? | 18:50 |
| *** Guest58148 has quit IRC | 18:50 | |
| nibalizer | pelix is probably already weekending | 18:51 |
| uberjay | fungi: sorry, to clarify, this in on my own CI, no zuul.o.o | 18:52 |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: networking-ovn: Allow gate hook scripts. https://review.openstack.org/186538 | 18:52 |
| *** mpaolino has joined #openstack-infra | 18:52 | |
| fungi | uberjay: i figured, just pointing out that no, we're not seeing that behavior in our zuul as far as i can tell | 18:53 |
| *** mpaolino has quit IRC | 18:53 | |
| mtreinish | fungi, mordred: have you guys seen this: https://jenkins01.openstack.org/job/check-tempest-dsvm-full/822/console it looks like the ansible run took like an hour | 18:53 |
| * mtreinish just realized that link will get less useful as the run continues... | 18:53 | |
| uberjay | fungi: k, i just wanted to see if it sounded familiar to anyone. hrm. | 18:53 |
| jeblair | nibalizer: oh cool hiera yaml is up. that slipped under my radar due to traveling etc. | 18:54 |
| *** jcoufal has joined #openstack-infra | 18:54 | |
| *** jcoufal has quit IRC | 18:55 | |
| fungi | mtreinish: um... wow! i missed that we had started using ansible in those jobs | 18:55 |
| mtreinish | fungi: I noticed it was running a while ago, but I'm not sure what it does :) | 18:56 |
| mtreinish | that was the first time I've seen it take ~1hr though | 18:56 |
| jeblair | its used in the multi-node jobs, and now in the single node jobs to make them use the same mech | 18:56 |
| fungi | oh, right. now i remember | 18:57 |
| fungi | i didn't realize it got run in single-node jobs too | 18:57 |
| fungi | but that makes sense | 18:57 |
| fungi | and yeah, i guess we want ansible to provide some additional progress output. what could it possible be doing for that 61-minute span of time? feels like something in there has a one-hour timeout | 18:58 |
| fungi | er, possibly | 18:58 |
| anteaya | fungi: the ansible playbooks I have seen have output turned off | 18:59 |
| jeblair | ftr the computer was operating during that time. :) there are entries in syslog. | 18:59 |
| *** mrmartin has joined #openstack-infra | 18:59 | |
| anteaya | would be easy to remove that and allow the playbooks to report their progress | 18:59 |
| jeblair | that's not running a playbook though, that's just invoking the shell module | 19:00 |
| anteaya | oh | 19:01 |
| jeblair | the only things it's doing are some shell tests, moving a directory (likely across filesystems) and symlinking | 19:01 |
| jeblair | 468K/opt/stack/logs/ | 19:01 |
| jeblair | and that's all the data it moved | 19:01 |
| jeblair | (likely less) | 19:02 |
| mgagne | is it possible to "smoothly" rename an IRC channel? | 19:02 |
| anteaya | mgagne: from what to what? | 19:02 |
| mgagne | anteaya: from an old name to a new name? :D | 19:02 |
| jeblair | mgagne: i think you can set up a forward (or maybe an irc staffer has to? i'm not sure) | 19:02 |
| jeblair | so that people join get automatically joined to the new channel | 19:02 |
| jeblair | i'm not sure that's smooth, but it's probably as smooth as it gets | 19:02 |
| anteaya | mgagne: my question was if both names include #openstack | 19:03 |
| anteaya | as jeblair has the capacity to ask for things in the #openstack namespace | 19:03 |
| mgagne | anteaya: the old name, if a pedantic regex is applied, not | 19:03 |
| mgagne | anteaya: the new name, yes | 19:03 |
| mgagne | now the actual use case: #puppet-openstack is backward compared to the other channel names :D | 19:04 |
| anteaya | oh well for puppet-openstack to be openstack-puppet | 19:04 |
| mgagne | yes | 19:04 |
| mgagne | that's the actual idea I had | 19:05 |
| *** geoffarnold has joined #openstack-infra | 19:05 | |
| anteaya | since openstack is in both I think there would be a case for freenode to listen if jeblair asked them | 19:05 |
| *** jgrimm is now known as zz_jgrimm | 19:05 | |
| nibalizer | a forward should work, the only issue is then people will type #puppet-openstack and never learn the new name | 19:05 |
| mgagne | I have no official request for it, just asking around about the feasibility | 19:05 |
| anteaya | mgagne: how many current users in the channel? | 19:06 |
| mgagne | nibalizer: is there a way to efficiently communicate this change? | 19:06 |
| anteaya | you could create the new channel and change the topic in puppet-openstack directing new folks to join openstack-puppet | 19:06 |
| geoffarnold | Having difficulty loading a review - https://review.openstack.org/%23/c/181125/ I get the "Loading Gerrit Review..." message, then nothing. I'm logged in to Launchpad | 19:06 |
| mgagne | currently 105 users | 19:06 |
| anteaya | mgagne: ah | 19:06 |
| anteaya | bit of a crowd | 19:06 |
| mgagne | I guess it's part of why puppet is popular with openstack =) | 19:07 |
| anteaya | geoffarnold: try https://review.openstack.org/#/c/181125 | 19:07 |
| nibalizer | mgagne: notices every four hours, and a resilent group of people grumping when conversation shows up in #puppet-openstack to move to #openstack-puppet | 19:08 |
| nibalizer | or have a bot grump | 19:08 |
| anteaya | geoffarnold: not sure where the %23 came from but # seems to work | 19:08 |
| nibalizer | really similar process to moving the email convos | 19:08 |
| mgagne | nibalizer: a bot that cripples any possible conversation in the old channel, asking to move to the new one? :D | 19:08 |
| geoffarnold | Thanks | 19:08 |
| nibalizer | these are suggestions comming from me, btw, not infra policy or anything | 19:08 |
| anteaya | geoffarnold: welcome | 19:08 |
| geoffarnold | WebEx mangling bug | 19:08 |
| anteaya | geoffarnold: yay WebEX | 19:09 |
| *** mwhagedorn has quit IRC | 19:09 | |
| *** otter768 has joined #openstack-infra | 19:09 | |
| anteaya | at least it never disapoints | 19:09 |
| nibalizer | mgagne: yea have it welcome new users and inform them of the change | 19:09 |
| openstackgerrit | Sahdev Zala proposed openstack-infra/project-config: Add #openstack-heat-translator to the list of irc channels https://review.openstack.org/186871 | 19:09 |
| mgagne | I get the idea, it's not technically impossible, you just need to do a lot of social migrations too | 19:09 |
| anteaya | mgagne: yes | 19:10 |
| *** _nadya_ has joined #openstack-infra | 19:10 | |
| *** crc32 has quit IRC | 19:10 | |
| nibalizer | also from some testing /mode ##spencertest +f ##spencertest2 doesn't do what you'd hope | 19:11 |
| mgagne | would there be any technical or "social" advantages to infra for the channel to be prefixed by #openstack? | 19:11 |
| adam_g | anteaya, any chance you can take a look here when you get a free sec ? https://review.openstack.org/#/c/186840/ forgot this in yesterdays patch | 19:12 |
| *** ajmiller has quit IRC | 19:12 | |
| *** ajmiller has joined #openstack-infra | 19:13 | |
| anteaya | mgagne: not for me personally, and not that I have been aware in past conversations | 19:13 |
| anteaya | adam_g: sure | 19:13 |
| *** otter768 has quit IRC | 19:13 | |
| anteaya | mgagne: I think the main issue was having meetbot log the channel which I do think was addressed yesterday, was it not? | 19:14 |
| mgagne | anteaya: yes, this part will soon be adressed | 19:14 |
| *** salv-orlando has joined #openstack-infra | 19:14 | |
| anteaya | mgagne: you have openstack in the channel name, which demonstrates to me the intent of the channel | 19:14 |
| anteaya | I don't feel we need to be logging channels that have nothing to do with openstack and are proprietary topic discussions about a plugin or driver | 19:15 |
| anteaya | but puppet-openstack tells me that your intent is to discuss puppet modules for openstack, so the order isn't important for me personally | 19:15 |
| anteaya | others may have differing opinions | 19:16 |
| *** freerunner has joined #openstack-infra | 19:16 | |
| fungi | mgagne: we moved #openstack-release to #openstack-stable some time back. i don't remember what steps i performed, but i seem to recall the ubuntu community had a pretty detailed writeup of the necessary steps | 19:16 |
| *** puranamr has quit IRC | 19:17 | |
| *** aduarte has quit IRC | 19:17 | |
| mgagne | fungi: IIRC, I got "kicked" of the channel by someone asking me to move to the new channel | 19:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: WIP: Create zuul-packaging experimental job https://review.openstack.org/186441 | 19:17 |
| anteaya | adam_g: +A | 19:18 |
| *** ihrachyshka has joined #openstack-infra | 19:18 | |
| adam_g | andreaf, thanks! | 19:18 |
| anteaya | he gets all my lauds | 19:18 |
| anteaya | :) | 19:18 |
| electrical | zaro: im back online decently ( as in not on my darn small phone :p ) see if i can make that change now regarding the error reporting | 19:19 |
| fungi | mgagne: yeah, i believe i had to add +f for myself, warned people for a few days, adjusted the topic, later set the old channel to private and set up a forwarder to the new one on attempts to join, then a few days after that kicked out any stragglers (along with a polite note) if they weren't paying attention | 19:20 |
| *** panda has quit IRC | 19:21 | |
| *** salv-orlando has quit IRC | 19:21 | |
| fungi | it took place slowly over the course of a couple weeks. there were also mailing list discussions about it in case anyone was paying closer attention to e-mail than irc | 19:21 |
| *** panda has joined #openstack-infra | 19:22 | |
| nibalizer | maybe the thing to do is register #openstack-puppet and have it forward into #puppet-openstack | 19:22 |
| nibalizer | that way if anyone is trying to follow the pattern they end up in the right place | 19:22 |
| fungi | mgagne: https://wiki.ubuntu.com/IRC/MovingChannels is roughly what i did | 19:23 |
| *** alexpilotti has quit IRC | 19:23 | |
| *** achanda_ has quit IRC | 19:23 | |
| fungi | with sufficient delays between steps | 19:24 |
| *** lnr has quit IRC | 19:24 | |
| *** ashleighfarnham has quit IRC | 19:26 | |
| mgagne | fungi: tyvm for your help, I will propose the idea to the community | 19:27 |
| *** samuelBartel has joined #openstack-infra | 19:27 | |
| electrical | zaro: is the MissingAttributeError a better error for the 'you forgot this mandatory setting' ? | 19:28 |
| *** soren has joined #openstack-infra | 19:29 | |
| *** freerunner has quit IRC | 19:30 | |
| electrical | zaro: if so, want me to submit a larger change to have all the plugins use the same? | 19:32 |
| *** koolhead17 has joined #openstack-infra | 19:34 | |
| *** yamahata has joined #openstack-infra | 19:36 | |
| *** salv-orlando has joined #openstack-infra | 19:38 | |
| *** timcline has quit IRC | 19:39 | |
| openstackgerrit | Merged openstack-infra/system-config: Enable logging of #puppet-openstack https://review.openstack.org/186521 | 19:40 |
| *** hashar has joined #openstack-infra | 19:40 | |
| anteaya | I'm trying to understand where ansible comes in on the tempest-dsvm-full job | 19:40 |
| anteaya | I'm looking at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n96 | 19:40 |
| *** salv-orlando has quit IRC | 19:41 | |
| anteaya | and have looked at the builder macros, they don't mention ansible | 19:41 |
| *** soren has quit IRC | 19:42 | |
| fungi | anteaya: it's in the source code of devstack-gate | 19:42 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/gear: Remove shebang as the script is managed by an entry_point https://review.openstack.org/186889 | 19:42 |
| fungi | git grep ansible in a checkout of openstack-infra/devstack-gate | 19:42 |
| openstackgerrit | Merged openstack-infra/project-config: Add relevant projects to $PROJECTS for akanda jobs https://review.openstack.org/186840 | 19:42 |
| anteaya | there it is: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n439 | 19:42 |
| *** yamahata has quit IRC | 19:43 | |
| *** sdake_ is now known as sdake | 19:43 | |
| anteaya | fungi: sorry first time I looked I had the name of the file wrong | 19:43 |
| *** yamahata has joined #openstack-infra | 19:43 | |
| *** koolhead_ has joined #openstack-infra | 19:43 | |
| fungi | no need to apologize! | 19:44 |
| *** dprince has quit IRC | 19:44 | |
| *** dannywilson has quit IRC | 19:44 | |
| pleia2 | anteaya: the files can be a bit tricky to drill down through :) | 19:44 |
| *** sdake_ has joined #openstack-infra | 19:46 | |
| pleia2 | fungi: can you do a quick review of https://review.openstack.org/#/c/182368/ so we can land this during our meetingless friday afternoon? | 19:47 |
| *** sridhar_ram has quit IRC | 19:47 | |
| *** koolhead17 has quit IRC | 19:47 | |
| *** HeOS has joined #openstack-infra | 19:48 | |
| Clint | that's going to merge-conflict the #kolla change | 19:48 |
| *** dannywilson has joined #openstack-infra | 19:48 | |
| *** patrickeast has joined #openstack-infra | 19:48 | |
| pleia2 | hrm, was hoping it would be smarter than that | 19:49 |
| Clint | maybe i'm wrong | 19:49 |
| *** sdake has quit IRC | 19:49 | |
| *** sdake has joined #openstack-infra | 19:49 | |
| fungi | pleia2: approve at will | 19:50 |
| pleia2 | ty | 19:50 |
| *** mrmartin has quit IRC | 19:50 | |
| fungi | Clint: looks like they're not within each others' context margins unless kolla's addition is out of alpha order | 19:50 |
| anteaya | pleia2: I'm seeing that, thanks for understanding :) | 19:50 |
| fungi | (i haven't looked) | 19:51 |
| anteaya | so from what I can see this is the last command that ran: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n449 | 19:51 |
| anteaya | then an hour passed | 19:51 |
| *** mrmartin has joined #openstack-infra | 19:51 | |
| anteaya | fungi: and thanks | 19:51 |
| *** ZZelle_ has quit IRC | 19:51 | |
| Clint | fungi: https://review.openstack.org/#/c/181673/6/modules/openstack_project/manifests/eavesdrop.pp,unified | 19:52 |
| sdake | fungi kolla problem? | 19:52 |
| fungi | Clint: oh, hrm yeah since it's not #openstack-kolla you're probably right | 19:52 |
| sdake | fungi do you need any help? | 19:52 |
| fungi | we can just rebase one on the other | 19:52 |
| *** timcline has joined #openstack-infra | 19:52 | |
| fungi | sdake: nah, just akanda and kolla fighting over merge conflicts in the eavesdrop channels list | 19:53 |
| *** timcline has quit IRC | 19:53 | |
| *** sdake_ has quit IRC | 19:53 | |
| sdake | fungi cool if someone is on it I'll let em to it:) | 19:53 |
| sdake | I think Clint was doing that magic :) | 19:53 |
| fungi | Clint: feel free to upload a rebase of 181673 on top of 182368 (or onto master once 182368 merges, whichever) | 19:54 |
| fungi | i'll happily approve | 19:54 |
| *** shashankhegde has quit IRC | 19:54 | |
| Clint | fungi: ok | 19:55 |
| pleia2 | Clint: thanks for keeping an eye on it, I thought I had found all the pending logging changes | 19:55 |
| Clint | np | 19:55 |
| pleia2 | friday search-fu is failing :) | 19:55 |
| fungi | pleia2: gertty master has file searching now | 19:55 |
| pleia2 | fungi: I should update! | 19:55 |
| fungi | no need to keep fumbling with the gerrit webui to perform changed file searches now | 19:56 |
| anteaya | and it looks like it was called here: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n483 | 19:56 |
| * pleia2 nods | 19:56 | |
| fungi | pleia2: just be prepared for a fairly lengthy sqlite migration to run when it's upgrading | 19:57 |
| fungi | mine took about 10-15 minutes to complete | 19:57 |
| pleia2 | fungi: aha, sounds like a task for lunch time | 19:57 |
| *** puranamr has joined #openstack-infra | 19:58 | |
| electrical | pleia2: and its always time for lunch :-) or was that beer o'clock? | 19:58 |
| fungi | also probably a good idea to back up ~/.gertty.db just in case | 19:58 |
| pleia2 | electrical: this week, they maybe the same thing | 19:59 |
| *** SumitNaiksatam has quit IRC | 19:59 | |
| electrical | its always 5 pm somewhere in the world right :-) | 19:59 |
| *** koolhead_ has quit IRC | 20:00 | |
| *** notnownikki has quit IRC | 20:00 | |
| openstackgerrit | Clint Adams proposed openstack-infra/system-config: Add #kolla to the list of statusbot/meetbot channels https://review.openstack.org/181673 | 20:00 |
| *** dannywilson has quit IRC | 20:00 | |
| Clint | fungi: pleia2: i think that was the right thing ^ | 20:01 |
| *** mjordan has joined #openstack-infra | 20:01 | |
| *** jerryz has quit IRC | 20:02 | |
| pleia2 | Clint: lgtm | 20:02 |
| pleia2 | openstack-puppet ended up in there in the meantime too, but I think it'll figure that one out | 20:03 |
| Clint | yeah, it got mad at me when i tried to do a more inclusive rebase | 20:03 |
| fungi | Clint: yep, note the "depends on" in its dependencies box below the votes table | 20:03 |
| *** SumitNaiksatam has joined #openstack-infra | 20:04 | |
| jgriffith | anybody reporting issues with git calls failing to "git.openstack.org"? | 20:04 |
| *** openstack has joined #openstack-infra | 20:06 | |
| mtreinish | jgriffith: probably... | 20:06 |
| mtreinish | I think that came up before, and was rejected | 20:06 |
| mtreinish | but don't hold me to that | 20:06 |
| fungi | right now i think we do them in our gerrit-git-prep job macro, and also in devstack-gate | 20:07 |
| jgriffith | mtreinish: nah, sure you're right | 20:07 |
| fungi | and there's some discussion about implementing similar retry logic in zuul-cloner too | 20:07 |
| *** sridhar_ram1 has joined #openstack-infra | 20:07 | |
| jgriffith | fungi: yeah, I'll put some logic in my ci runner to do that as well | 20:07 |
| jgriffith | fungi: thanks for the info | 20:07 |
| jgriffith | mtreinish: and thanks to you for saving me the rejection of another devstack patch :) | 20:07 |
| fungi | you should feel free to crib the retry routines we already have | 20:07 |
| jgriffith | fungi: yep, that's my plan :) | 20:08 |
| jgriffith | ./clean.sh | 20:08 |
| jgriffith | crap | 20:08 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: drop support for alpha releases https://review.openstack.org/186902 | 20:08 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: create missing milestones when releasing libraries https://review.openstack.org/186903 | 20:08 |
| *** sridhar_ram has quit IRC | 20:08 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/devstack-gate: Make ansible more verbose in gate-wrap https://review.openstack.org/186900 | 20:09 |
| anteaya | mtreinish: support my patch for chatty ansible output ^^ | 20:09 |
| *** fallenpegasus has quit IRC | 20:10 | |
| openstackgerrit | Merged openstack-infra/system-config: Add #akanda to openstack bot's channel list https://review.openstack.org/182368 | 20:10 |
| mtreinish | anteaya: was that a question or an order? :) | 20:11 |
| *** ashleighfarnham has joined #openstack-infra | 20:11 | |
| *** SumitNaiksatam has quit IRC | 20:11 | |
| anteaya | mtreinish: an encouragement | 20:11 |
| *** SumitNaiksatam has joined #openstack-infra | 20:11 | |
| anteaya | like sign my petition | 20:11 |
| mtreinish | heh, ok | 20:12 |
| anteaya | :) | 20:12 |
| openstackgerrit | Geoff Arnold proposed openstack-infra/project-config: Add new Stackforge project Mercador This is commit 1 of 3 for the project. Mercador is a project for OpenStack service federation, building on the Keystone work in Kilo. The project is described at https://wiki.openstack.org/wiki/Mercador https://review.openstack.org/186906 | 20:14 |
| fungi | wow that's more than 50 characters | 20:15 |
| geoffarnold | oops | 20:15 |
| *** fijal has quit IRC | 20:16 | |
| anteaya | and welcome geoffarnold | 20:17 |
| *** soren has joined #openstack-infra | 20:18 | |
| sdake | fungi lol missing CR Ftw | 20:19 |
| openstackgerrit | Geoff Arnold proposed openstack-infra/project-config: Add Stackforge project Mercador https://review.openstack.org/186906 | 20:20 |
| *** _ody has quit IRC | 20:20 | |
| *** freerunner has joined #openstack-infra | 20:20 | |
| *** mrmartin has quit IRC | 20:20 | |
| *** _ody has joined #openstack-infra | 20:21 | |
| *** Sukhdev has quit IRC | 20:21 | |
| electrical | zaro: " jenkins_jobs.errors.MissingAttributeError: Missing room from an instance of publisher.slack " <-- don't really like this error message tbh | 20:21 |
| electrical | the others aren't much better either. | 20:22 |
| openstackgerrit | Geoff Arnold proposed openstack-infra/project-config: Add Stackforge project Mercador https://review.openstack.org/186906 | 20:22 |
| *** freerunner has quit IRC | 20:22 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 20:23 | |
| electrical | zaro: making a custom error looks much better: jenkins_jobs.errors.JenkinsJobsException: Missing 'room' option in Slack publisher | 20:24 |
| electrical | zaro: perhaps we can address it as part of the rewrite to improve it | 20:24 |
| electrical | make the type of errors much clearer | 20:24 |
| *** mdrnstm has joined #openstack-infra | 20:28 | |
| *** mdrnstm has quit IRC | 20:28 | |
| *** mdrnstm has joined #openstack-infra | 20:28 | |
| waynr | electrical: we should definitely talk about that, but as far as the rewrite that i've been talking about goes I have only been referring to rewriting the YamlParser and related yaml loading classes | 20:29 |
| *** openstack has joined #openstack-infra | 20:31 | |
| waynr | we could just "from jenkins_jobs import errors" then reference exceptions as "errors.JenkinsJobsException" but to be honest that feels too nit picky | 20:32 |
| openstackgerrit | Richard Pijnenburg proposed openstack-infra/jenkins-job-builder: Added slack support https://review.openstack.org/131404 | 20:32 |
| electrical | waynr: yeah indeed | 20:32 |
| electrical | we should be able to refer to it like its now but have the import a bit simpler. | 20:32 |
| *** mjordan has left #openstack-infra | 20:34 | |
| *** sdake_ has joined #openstack-infra | 20:36 | |
| *** pc_m has quit IRC | 20:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Add #openstack-heat-translator to the list of irc channels https://review.openstack.org/186871 | 20:37 |
| electrical | waynr: i think some things we can do independent of the yaml rewrite. like re-designing the type of errors and textual context of them | 20:37 |
| *** Swami has quit IRC | 20:39 | |
| *** sdake has quit IRC | 20:40 | |
| *** gokrokve has quit IRC | 20:40 | |
| zaro | electrical: the msg seems ok to me but then again i might have been reviewing jjb too long :) | 20:40 |
| *** alexpilotti has joined #openstack-infra | 20:41 | |
| electrical | zaro: hehe. i did push a new change to make the error slightly nicer then it was but still using the JenkinsJobsException | 20:41 |
| *** fallenpegasus has joined #openstack-infra | 20:41 | |
| waynr | zaro: i will definitely continue keeping my eye out for disruptive core changes | 20:41 |
| zaro | waynr: dude, we are counting on you to just fix it all :) | 20:41 |
| *** annegentle has joined #openstack-infra | 20:42 | |
| *** fallenpegasus has quit IRC | 20:43 | |
| electrical | no pressure waynr haha. | 20:44 |
| *** fallenpegasus has joined #openstack-infra | 20:44 | |
| *** vahidh has quit IRC | 20:44 | |
| *** fallenpegasus has quit IRC | 20:45 | |
| *** vahidh has joined #openstack-infra | 20:46 | |
| *** gokrokve has joined #openstack-infra | 20:46 | |
| waynr | :) i'm about to go silent for the rest of the day, looking forward to catching up with reviews and specs over the weekend | 20:47 |
| openstackgerrit | Richard Pijnenburg proposed openstack-infra/jenkins-job-builder: Added slack support https://review.openstack.org/131404 | 20:47 |
| electrical | waynr: i should be online most of the weekend if you want to bounce idea's / discussions. | 20:47 |
| *** dannywilson has joined #openstack-infra | 20:47 | |
| *** mattfarina has quit IRC | 20:48 | |
| *** spzala has quit IRC | 20:49 | |
| *** spzala has joined #openstack-infra | 20:49 | |
| *** shashankhegde has joined #openstack-infra | 20:49 | |
| *** linuxmaniac has quit IRC | 20:49 | |
| *** peristeri has quit IRC | 20:49 | |
| *** sdake has joined #openstack-infra | 20:50 | |
| *** SumitNaiksatam has quit IRC | 20:50 | |
| *** nelsnelson has quit IRC | 20:50 | |
| *** gokrokve has quit IRC | 20:50 | |
| *** koolhead17 has joined #openstack-infra | 20:51 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:53 | |
| electrical | zaro: got about 8 changes that should be okay to kick further up the chain ( some of them have a +2 from you already ) | 20:54 |
| *** sdake_ has quit IRC | 20:54 | |
| *** linuxmaniac has joined #openstack-infra | 20:54 | |
| *** fijal has joined #openstack-infra | 20:56 | |
| openstackgerrit | Merged openstack-infra/system-config: Add #kolla to the list of statusbot/meetbot channels https://review.openstack.org/181673 | 20:56 |
| *** puranamr has quit IRC | 20:56 | |
| *** dguerri`away is now known as dguerri | 20:56 | |
| *** puranamr_ has joined #openstack-infra | 20:57 | |
| *** puranamr has joined #openstack-infra | 20:57 | |
| *** _nadya_ has quit IRC | 20:57 | |
| *** sridhar_ram has joined #openstack-infra | 20:59 | |
| *** openstackgerrit has quit IRC | 20:59 | |
| *** openstackgerrit has joined #openstack-infra | 20:59 | |
| zaro | electrical: our counts don't match up. i only see about 2 +2 that i haven't reviewed. | 21:00 |
| electrical | zaro: i was very selective ;-) | 21:01 |
| zaro | electrical: i see, you meant +1. | 21:01 |
| *** pradk has quit IRC | 21:02 | |
| electrical | want me to link them somewhere so you can take a peek? | 21:02 |
| *** puranamr_ has quit IRC | 21:02 | |
| *** puranamr has quit IRC | 21:02 | |
| openstackgerrit | Merged openstack/requirements: Update sqlalchemy-migrate to 0.9.6 https://review.openstack.org/181821 | 21:02 |
| zaro | electrical: btw using JenkinsJobException is ok with me | 21:02 |
| electrical | okay :-) | 21:03 |
| openstackgerrit | Merged openstack/requirements: Bump oslo.vmware version to 0.13.1 https://review.openstack.org/186574 | 21:03 |
| zaro | electrical: nah, i see them. just need more time :) | 21:03 |
| electrical | hehe okay | 21:03 |
| electrical | time is always against us | 21:04 |
| *** fallenpegasus has joined #openstack-infra | 21:04 | |
| *** fawadkhaliq has quit IRC | 21:05 | |
| *** gokrokve has joined #openstack-infra | 21:07 | |
| *** julim has quit IRC | 21:07 | |
| openstackgerrit | Merged openstack/requirements: Bump ddt to min version 0.7.0 https://review.openstack.org/166162 | 21:08 |
| *** dpaterson has quit IRC | 21:08 | |
| electrical | time for me to grab some sleep. catch you all after the weekend. | 21:08 |
| *** otter768 has joined #openstack-infra | 21:09 | |
| mattt | lifeless: had an unrelated pip question which i thought you may have been able to answer, but figured it out in the end :P | 21:10 |
| *** bswartz has quit IRC | 21:10 | |
| *** Swami has joined #openstack-infra | 21:13 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:14 | |
| clarkb | fungi: mordred: I am back for the afternoon, do we have ubuntu-trusty in rax yet? | 21:14 |
| * clarkb should dig up his key and ssh in to check | 21:14 | |
| *** otter768 has quit IRC | 21:15 | |
| *** hashar has quit IRC | 21:15 | |
| fungi | clarkb: not sure... image builds may have been aborted during the nodepool troubleshooting this morning | 21:15 |
| *** hashar has joined #openstack-infra | 21:15 | |
| clarkb | good point | 21:15 |
| clarkb | I will try a manual upload if none are in progress | 21:15 |
| fungi | i put my work depending on new images on the back burner and figure i'll pick it back up after image updates tomorrow or perhaps monday | 21:15 |
| fungi | switched to different tasks for the afternoon | 21:16 |
| clarkb | ya I see you busy on the mail list | 21:16 |
| fungi | there was a lot i had flagged that needed replies | 21:16 |
| fungi | i still have a few to get to | 21:16 |
| *** annegentle has quit IRC | 21:17 | |
| *** jsavak has quit IRC | 21:17 | |
| *** ldnunes has quit IRC | 21:18 | |
| clarkb | I have mostly declared ml debt at this point | 21:18 |
| clarkb | so thank you for keeping up | 21:18 |
| clarkb | looks like uploads are in progress and a couple just started. makes me wonder if there is a recurring error | 21:19 |
| fungi | i honestly didn't expect to see you around (still think you probably shouldn't be!) | 21:19 |
| clarkb | well larissa is basically confined to couch/bed which leaves me with house duties or hacking | 21:20 |
| clarkb | I will probably disappear once twins arrive | 21:20 |
| fungi | i certainly hope so. priorities | 21:21 |
| *** koolhead17 has quit IRC | 21:21 | |
| lifeless | mattt: heh, ok :) | 21:21 |
| asselin | clarkb, congrats...fun stuff I see :) | 21:21 |
| lifeless | mattt: btw there is #pypa-dev (and perhaps #pypa?) which are dedicated to pip and related stuff | 21:21 |
| *** amitgandhinz has quit IRC | 21:22 | |
| *** sdake is now known as steakache | 21:23 | |
| *** soren has quit IRC | 21:24 | |
| *** steakache is now known as sdake | 21:24 | |
| *** salv-orlando has joined #openstack-infra | 21:26 | |
| clarkb | "413 Request Entity Too Large" thats the current issue | 21:26 |
| fungi | clarkb: are we getting that from swift during upload? | 21:27 |
| clarkb | notmyname: ^ does swiftclient need to be told how to do split up objects for large objects or will it do that for us? | 21:27 |
| notmyname | clarkb: SDK or CLI? | 21:27 |
| clarkb | fungi: ya, our images are 14GB but swift allows up to 5GB objects in rax iirc so we have to do the swift large object stuff | 21:27 |
| clarkb | notmyname: python-swiftclient | 21:27 |
| notmyname | clarkb: which has both a CLI and SDK ;-) | 21:27 |
| notmyname | are you doing command line stuff or python? | 21:28 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: update openstackclient requirement to 1.0.3 https://review.openstack.org/186919 | 21:28 |
| mordred | notmyname: python | 21:28 |
| clarkb | notmyname: oh I thought you were asking openstacksdk vs swiftclient, its python sdk in swiftclient not cli | 21:28 |
| *** krtaylor has quit IRC | 21:28 | |
| mordred | clarkb: we should also make the images stop being 14G - but that's another thing | 21:28 |
| notmyname | let me check. I know the CLI has some auto-splitting functionality. let me see how/if that's plumbed through to the SDK | 21:29 |
| clarkb | mordred: yes :) | 21:29 |
| *** shashankhegde has quit IRC | 21:29 | |
| fungi | like, see if rax's glance will autoconvert from qcow2 to whatever it needs to be able to boot a xen domu? | 21:30 |
| clarkb | fungi: thats one way to do it ya | 21:30 |
| clarkb | glance apparently supports this now | 21:30 |
| mordred | well ... | 21:30 |
| mordred | it supports the framework | 21:30 |
| *** openstack has joined #openstack-infra | 21:32 | |
| anteaya | mordred: how many -v's should ansible have to tell us why it takes an hour to move a folder? https://review.openstack.org/#/c/186900/ | 21:33 |
| anteaya | odyssey4me: feels 3 is too many | 21:33 |
| mordred | anteaya: maybe 4? | 21:33 |
| * mordred would only know by trying | 21:33 | |
| anteaya | well 4 is more than 3 | 21:34 |
| notmyname | clarkb: mordred: if you're using the low-level client.py interface (or a Connection object), then you'll need to do the splitting yourself. if you're using the higher-level SwiftService object, then that can take care of it for you (and make associated manifest objects) | 21:34 |
| anteaya | and yes, I have no idea until we give it a whirl | 21:34 |
| SpamapS | I doubt 10 v's will tell you why that mv is slow | 21:35 |
| SpamapS | but it's possible ssh'ing is going really slow | 21:35 |
| mordred | notmyname: nod | 21:36 |
| SpamapS | which would probably need 4 v's to debug | 21:36 |
| *** sridhar_ram has quit IRC | 21:36 | |
| mordred | notmyname: thanks! we're using the client.py/Connection right now - looks like I should look in to the SwiftService object | 21:36 |
| clarkb | notmyname: thank your for looking | 21:36 |
| anteaya | SpamapS: mind adding a comment to the patch? | 21:37 |
| anteaya | SpamapS: I don't care if I get the number of v's wrong in the first try, I would just like to have more that we have now | 21:37 |
| notmyname | mordred: clarkb: yeah joel added the service thing a while back so that SDK users could take advantage of the same functionality that was available in the CLI (and also improve testing/DRY/modularity) | 21:37 |
| mordred | SpamapS, Shrews: ^^ if one of you beat me to this - looks like we may need to use SwiftService for swift uploads | 21:38 |
| SpamapS | anteaya: also why is that using the shell module? | 21:39 |
| clarkb | SpamapS: thats intentional iirc | 21:39 |
| *** fallenpegasus has quit IRC | 21:39 | |
| clarkb | oh no thats all copy and file so not sure what you refer to | 21:40 |
| SpamapS | a playbook using the file module would likely be more efficient and provide more structured information than -v with the shell module. | 21:40 |
| *** tonytan4ever has quit IRC | 21:40 | |
| *** salv-orlando has quit IRC | 21:40 | |
| clarkb | SpamapS: we are intentionally not using playbooks | 21:40 |
| clarkb | mordred wrote a change to use playbooks and it throws off the readability significantly | 21:40 |
| clarkb | with all of the control flow and logic in bash it makes a lot of sense to use ansible on a per command basis | 21:41 |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: Rename stackforge/chef* to openstack/chef* https://review.openstack.org/186532 | 21:41 |
| clarkb | if/when we move the control flow and config into ansible playbooks make senes | 21:41 |
| anteaya | SpamapS: just trying to solve the issue in the log linked in the commit message | 21:42 |
| anteaya | what happened in that missing hour? | 21:42 |
| clarkb | anteaya: we have seen networking between nodes fail occasionally, but not between the main node and jenkins | 21:43 |
| SpamapS | wait.. the job I saw, and the one that's being modified, are very different | 21:43 |
| SpamapS | the one being modified is doing what I would have done w/ a playbook | 21:43 |
| anteaya | clarkb: I think if we go with 3 v's we get networking connection info as well | 21:43 |
| anteaya | SpamapS: the job I saw, what job is that? | 21:44 |
| SpamapS | anteaya: the log linked is doing -m shell. The job modified is doing -m copy and -m file.. ?? | 21:44 |
| clarkb | anteaya: reading that job log its taking all its time in setup workspace | 21:44 |
| clarkb | anteaya: so to debug that we should look at the setup workspace log | 21:44 |
| anteaya | have I modified the wrong file? | 21:44 |
| clarkb | SpamapS: yes I don't think the modifications touch where we lost an hour | 21:44 |
| *** doug-fish has left #openstack-infra | 21:44 | |
| SpamapS | clarkb: agree | 21:44 |
| clarkb | http://logs.openstack.org/70/170270/8/check/check-tempest-dsvm-full/c794401/logs/devstack-gate-setup-workspace-new.txt.gz | 21:45 |
| clarkb | and we specifically use shell there instead of a playbook because we aren't rewriting all of d-g in ansible any time soon | 21:45 |
| SpamapS | Yeah, I see all of that now for what it is. :) | 21:45 |
| *** fallenpegasus has joined #openstack-infra | 21:45 | |
| SpamapS | clarkb: the file being modified is at least doing things with -m file, which makes me feel a little better. :) | 21:46 |
| clarkb | what ansible does give us here is an effective way to run existing d-g on multiple nodes with error checking and parallel execution | 21:46 |
| jlvillal | sdague: What IRC channel do you discuss gerrit-dash-creator? | 21:46 |
| *** annegentle has joined #openstack-infra | 21:46 | |
| clarkb | jlvillal: I think sdague is out this week | 21:46 |
| *** samuelBartel has quit IRC | 21:46 | |
| jlvillal | clarkb: Okay thanks! | 21:46 |
| *** ddieterly has quit IRC | 21:47 | |
| clarkb | SpamapS: as part of the zuulv3 work (likely in parallel or nowish even) we will probably rewrite chunks of d-g at a time to be more ansibly | 21:48 |
| clarkb | and move things into playbook(s) | 21:48 |
| *** aysyanne has quit IRC | 21:48 | |
| anteaya | that took 15 minutes: http://logs.openstack.org/70/170270/8/check/check-tempest-dsvm-full/c794401/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-05-29_18_27_39_300 | 21:48 |
| SpamapS | clarkb: yeah, in originally suggesting it I was more thinking it would be helpful in debugging the issue at hand. But not worth the structural changes necessary. Though I do wonder if an inline playbook would keep the flow readable, I don't know if such a thing has drawbacks I haven't considered. | 21:50 |
| anteaya | the git fetches seem to be taking some time | 21:50 |
| clarkb | SpamapS: thats a good question, my gripe with the ealier playbook change was it pulled a lot of the what and hid it, inlining the playbooks would probably fix that | 21:50 |
| clarkb | SpamapS: and we would reduce the number of connections too, that is probably work looking at | 21:51 |
| anteaya | this one took over 10 minutes: http://logs.openstack.org/70/170270/8/check/check-tempest-dsvm-full/c794401/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-05-29_18_01_47_922 | 21:51 |
| *** mriedem has quit IRC | 21:51 | |
| *** salv-orlando has joined #openstack-infra | 21:52 | |
| *** marcusvrn has quit IRC | 21:53 | |
| *** dims_ has joined #openstack-infra | 21:53 | |
| *** akshai has quit IRC | 21:53 | |
| *** stevemar has quit IRC | 21:53 | |
| anteaya | so my patch isn't useful and I'll abandon | 21:53 |
| *** dimtruck is now known as zz_dimtruck | 21:53 | |
| dstufft | dhellmann: fwiw PEP 440 supports epochs too | 21:54 |
| *** mdenny has quit IRC | 21:54 | |
| *** dimsum__ has quit IRC | 21:55 | |
| *** alexpilotti has quit IRC | 21:56 | |
| *** fallenpegasus has quit IRC | 21:57 | |
| *** tim_o has joined #openstack-infra | 22:00 | |
| *** hashar has quit IRC | 22:01 | |
| *** geoffarnold has quit IRC | 22:04 | |
| *** fawadkhaliq has joined #openstack-infra | 22:05 | |
| jeblair | AJaeger: somehow i am not surprised that you are the first to leave an infra-council vote on a change ;) | 22:05 |
| *** claudiub has quit IRC | 22:06 | |
| *** ashleighfarnham has quit IRC | 22:06 | |
| mordred | jeblair: wait! where? I wanna see it... | 22:09 |
| anteaya | https://review.openstack.org/#/c/184559/ | 22:10 |
| *** fawadkhaliq has quit IRC | 22:11 | |
| sigmavirus24 | do we know what the locale settings are on the jenkins boxes? http://logs.openstack.org/22/186922/1/check/gate-nova-python34/b40367b/console.html#_2015-05-29_21_53_34_135 shouldn't be happening if the locale is correct | 22:11 |
| clarkb | sigmavirus24: tox 2.0 filters it out | 22:11 |
| clarkb | sigmavirus24: so C is used as python default iirc | 22:12 |
| sigmavirus24 | clarkb: https://review.openstack.org/#/c/186922/ | 22:12 |
| sigmavirus24 | (which is where that log is from) | 22:12 |
| sigmavirus24 | So it should be passed unless passenv doesn't take LC_* (which I thought it did =/) | 22:12 |
| clarkb | oh in that case its en_US.UTF-8 iirc | 22:12 |
| clarkb | you might also need to pass LANGUAGE? | 22:12 |
| clarkb | ya Ithink it globs | 22:13 |
| sigmavirus24 | LANGUAGE is hardcoded above to en_US | 22:13 |
| clarkb | ah | 22:13 |
| sigmavirus24 | ah, but I guess, that's incomplete | 22:13 |
| clarkb | no LANGUAGE drops the encoding iirc | 22:13 |
| sigmavirus24 | okay | 22:13 |
| clarkb | it does on my laptop | 22:13 |
| fungi | en_DK.UTF-8 ftw! | 22:13 |
| sigmavirus24 | fungi: :D | 22:13 |
| sigmavirus24 | what's this Donkey Kong encoding? =P | 22:13 |
| *** _ody is now known as ody | 22:14 | |
| fungi | i actually use C.UTF-8 for sort, en_DK.UTF-8 for date/time and units, en_US.UTF-8 for the rest | 22:14 |
| sigmavirus24 | Yeah I'm kind of baffled by this gate problem | 22:14 |
| jeblair | mordred: i wonder if it's better to have green-check/red-x or +/-1 .... (i think we could re-expand the code-review category but just give no one permission to vote in the higher magnitude values | 22:14 |
| sigmavirus24 | the text causing that error has been in the package since 0.1.0 and we've been using >= 0.2.0 for almost a year now | 22:14 |
| mordred | jeblair: hrm. that's a good question. the green check certainly did cause me to assume a thing | 22:15 |
| jeblair | yep, and the red-x moreso for me | 22:16 |
| jeblair | (it looks a bit like pleia2 has vetoed the change, but that's not the case) | 22:16 |
| mordred | yah | 22:16 |
| pleia2 | yeah, just a -1 | 22:16 |
| *** ody is now known as _ody | 22:16 | |
| jeblair | i think i want to change it; i'll write up a patch in a bit | 22:17 |
| mordred | I think maybe expand both and give the chair +2 powers similar to ttx? or maybe something else? or I don't know | 22:17 |
| jeblair | mordred: nah, we can give no-one +2 powers and it will be ok | 22:17 |
| jeblair | we can expand the range from -2 to +2, give only -1 to +1 perms and since it's a noop function, it will be fine | 22:17 |
| *** ihrachyshka has quit IRC | 22:17 | |
| jlvillal | sigmavirus24: I ran into a similar issue in python-ironicclient. Ending up doing this: https://review.openstack.org/#/c/183339/ | 22:18 |
| mordred | jeblair: I believe you | 22:18 |
| *** annegentle has quit IRC | 22:18 | |
| sigmavirus24 | jlvillal: thanks. Let me try that too | 22:18 |
| jlvillal | sigmavirus24: Not sure if it would help you or not. | 22:18 |
| jlvillal | sigmavirus24: And we also had the situation where things were working and then sometimes it would fail. So it seems like systems are not configured exactly the same. | 22:19 |
| clarkb | jlvillal: oh interesting. LC_ALL is unset on my laptop | 22:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Enable trove plugin for trove tempest job https://review.openstack.org/168145 | 22:19 |
| openstackgerrit | Merged openstack-infra/project-config: Alter governance repo voting rules https://review.openstack.org/185785 | 22:19 |
| *** bdemers has quit IRC | 22:19 | |
| sigmavirus24 | jlvillal: not all jenkins boxes are equivalent ;) | 22:20 |
| clarkb | ya it oculd also be a difference between rax and hpcloud as base image is different | 22:20 |
| clarkb | (we are so close to fixing that though) | 22:20 |
| jlvillal | sigmavirus24: I think so. | 22:20 |
| jlvillal | clarkb: That would be great if they were identical :) | 22:20 |
| *** annegentle has joined #openstack-infra | 22:20 | |
| sigmavirus24 | clarkb: until then | 22:21 |
| greghaynes | clarkb: what is the state of that btw? | 22:21 |
| sigmavirus24 | let's see if jlvillal's fix works for nova | 22:21 |
| jlvillal | sigmavirus24: Looks your failure is on hpcloud | 22:21 |
| greghaynes | clarkb: we fixed the params issue and havent retried? | 22:21 |
| sigmavirus24 | jlvillal: let me check the other log | 22:21 |
| jlvillal | sigmavirus24: So make sure if you try again and it passes that it also passes on hpcloud | 22:21 |
| *** shashankhegde has joined #openstack-infra | 22:21 | |
| sigmavirus24 | jlvillal: yeah, the original bug was generated from http://logs.openstack.org/15/186315/4/check/gate-nova-python34/c48b0f5/console.html#_2015-05-29_15_57_16_241 which was hpcloud as well | 22:22 |
| clarkb | greghaynes: we don't have proper large object support in shade | 22:23 |
| greghaynes | oh? | 22:23 |
| clarkb | greghaynes: and 14GB image is >> object size limit | 22:23 |
| jlvillal | sigmavirus24: Okay. Well hopefully your test will also run on hpcloud. | 22:23 |
| greghaynes | hah | 22:23 |
| greghaynes | awesome | 22:23 |
| sigmavirus24 | jlvillal: *crosses fingers* | 22:23 |
| sigmavirus24 | if not, I'll recheck until it does | 22:23 |
| greghaynes | is anyone hacking on fixing that? | 22:23 |
| greghaynes | if not I can poke at it | 22:24 |
| greghaynes | mordred: ^ | 22:24 |
| mordred | greghaynes: not yet | 22:24 |
| clarkb | greghaynes: notmyname describes the fix in scrollback | 22:25 |
| mordred | greghaynes: the answer is apparently to use SwiftService | 22:25 |
| clarkb | there is a higher level interface we can use ^ | 22:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Expand the vote range for infra-specs https://review.openstack.org/186934 | 22:25 |
| mordred | the threading model will probably be weird with TaskManager - but for the action that's actually probably fineish? | 22:25 |
| jeblair | mordred, anteaya: ^ | 22:25 |
| greghaynes | Ah, I see in SB | 22:26 |
| * anteaya looks | 22:26 | |
| *** hdd has quit IRC | 22:27 | |
| *** armax has quit IRC | 22:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Expand the vote range for governance https://review.openstack.org/186937 | 22:28 |
| *** armax has joined #openstack-infra | 22:29 | |
| *** ivar-laz_ has joined #openstack-infra | 22:31 | |
| asselin | I'm testing out a new ci setup. Zull has the jobs 'queued', but nothing is being sent to jenkins? tail -f /var/log/jenkins/jenkins.log is empty when invoking another job | 22:33 |
| clarkb | asselin: check if you have a merger running | 22:34 |
| *** ivar-lazzaro has quit IRC | 22:34 | |
| asselin | clarkb, I did not. now it's working.glad I asked! :) | 22:34 |
| *** Cutie has joined #openstack-infra | 22:35 | |
| *** Cutie has quit IRC | 22:36 | |
| *** krtaylor has joined #openstack-infra | 22:36 | |
| *** hemna is now known as hemnabeer | 22:36 | |
| *** sdake_ has joined #openstack-infra | 22:39 | |
| *** annegentle has quit IRC | 22:40 | |
| *** sdake has quit IRC | 22:43 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Test that we are setting region-name properly https://review.openstack.org/186944 | 22:44 |
| greghaynes | clarkb: ^ that catches the bug from earlier | 22:45 |
| *** samueldmq has quit IRC | 22:47 | |
| *** samueldmq has joined #openstack-infra | 22:48 | |
| *** boris-42 has quit IRC | 22:48 | |
| *** puranamr has joined #openstack-infra | 22:48 | |
| *** puranamr_ has joined #openstack-infra | 22:48 | |
| *** puranamr_ has quit IRC | 22:49 | |
| *** puranamr has quit IRC | 22:49 | |
| *** sdake has joined #openstack-infra | 22:49 | |
| *** sdake has quit IRC | 22:49 | |
| *** baoli_ has quit IRC | 22:49 | |
| *** sdake has joined #openstack-infra | 22:49 | |
| *** baoli has joined #openstack-infra | 22:49 | |
| sigmavirus24 | jlvillal: thanks again. that fixed it | 22:50 |
| jlvillal | sigmavirus24: Sweet! :) | 22:50 |
| jlvillal | sigmavirus24: I wonder what LC_ALL is set to on hpcloud? | 22:50 |
| sigmavirus24 | jlvillal: if it's unset then Py3.4 will default to C I bet | 22:50 |
| jlvillal | sigmavirus24: Maybe | 22:51 |
| jlvillal | sigmavirus24: I wonder if it differs between rax and hpcloud. Probably does. | 22:51 |
| * sigmavirus24 shrugs | 22:51 | |
| sigmavirus24 | I can check but I'm lazy | 22:51 |
| sigmavirus24 | and it's friday night =P | 22:51 |
| jlvillal | :) | 22:51 |
| *** sdake_ has quit IRC | 22:52 | |
| sigmavirus24 | hm | 22:53 |
| jlvillal | Can anyone explain what OpenStack proposal bot is doing with this patch? https://review.openstack.org/#/c/128965/ | 22:53 |
| sigmavirus24 | it isn't set on my rax cloud box or on my local machine | 22:53 |
| jlvillal | clarkb: Maybe you^^^ https://review.openstack.org/#/c/128965/ | 22:53 |
| clarkb | jlvillal: ya | 22:53 |
| jlvillal | clarkb: Not sure if it should be abandoned? | 22:53 |
| clarkb | jlvillal: no do not abandon | 22:54 |
| sigmavirus24 | jlvillal: perhaps someone who isn't OpenStack Proposal Bot? | 22:54 |
| clarkb | clearly we need better commit messages on those changes | 22:54 |
| jlvillal | clarkb: I don't have the power to do it. | 22:54 |
| clarkb | jlvillal: what it is doing is important | 22:54 |
| jlvillal | clarkb: Okay. It is just sitting there in purgatory it seems. | 22:54 |
| clarkb | jlvillal: when you make a relase on a branch that isn't master, the master branch is unaware of the release happening which makes pbr's versioning stuff not work quite right | 22:54 |
| clarkb | jlvillal: so to put the release in master's history we merge the release back into master without mergingany code changes in | 22:55 |
| *** emagana has quit IRC | 22:55 | |
| *** emagana has joined #openstack-infra | 22:55 | |
| clarkb | so its a merge strategy of ours from masters perspective. The functional effect is that master can go backwards and get a path in the tree to the release | 22:55 |
| jlvillal | clarkb: Okay. Makes sense. A better commit message would probably be good to explain that for the reviewers. | 22:56 |
| clarkb | also lol that that never merged | 22:56 |
| clarkb | mordred: ttx ^ you likely need to better educate core teams and maybe make that commit message informative | 22:56 |
| jlvillal | clarkb: Are you able to kick the job? Seems to be in merge conflict | 22:56 |
| clarkb | jlvillal: someone will have to push an update, release manager ost likely | 22:56 |
| jlvillal | clarkb: I am not core team. So maybe it is just me who is clueless :) | 22:57 |
| clarkb | jlvillal: no that shouldve merged a half a year ago | 22:57 |
| *** ashleighfarnham has joined #openstack-infra | 22:57 | |
| jeblair | if the proposal bot isn't going to stick around to answer questions, it really ought to leave a good message the first time around :) | 22:57 |
| clarkb | but instead of asking question or pointing out the confusion it just got -1s "I don't understand" | 22:57 |
| clarkb | jeblair: ++ | 22:57 |
| jlvillal | jeblair: +1 :) | 22:58 |
| *** mdrnstm has quit IRC | 23:02 | |
| *** baoli has quit IRC | 23:04 | |
| *** SumitNaiksatam has quit IRC | 23:04 | |
| *** sputnik13 has quit IRC | 23:04 | |
| *** baoli has joined #openstack-infra | 23:05 | |
| *** salv-orlando has quit IRC | 23:07 | |
| *** otter768 has joined #openstack-infra | 23:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Expand the vote range for infra-specs https://review.openstack.org/186934 | 23:15 |
| *** otter768 has quit IRC | 23:15 | |
| *** dangers is now known as dangers_away | 23:17 | |
| fungi | probably also educating people though that leaving a negative review and expecting the proposal bot to reply explaining itself is not a workflow likely to produce results | 23:17 |
| fungi | the proposal bot is a fire-and-forget contributor of the worst kind | 23:18 |
| *** baoli has quit IRC | 23:18 | |
| jroll | I mean, that -1 led to someone asking about it here, and leaving a paste.o.o about it | 23:18 |
| jroll | and subsequently a +W | 23:18 |
| clarkb | fungi well in theory its not supposed to get -1s | 23:19 |
| clarkb | if it does thats a bug | 23:19 |
| clarkb | in thid case need to document in commit message what is happening | 23:20 |
| *** RichardRaseley has joined #openstack-infra | 23:24 | |
| *** tjones1 has quit IRC | 23:25 | |
| fungi | absolutely | 23:29 |
| *** wenlock has quit IRC | 23:30 | |
| *** baoli has joined #openstack-infra | 23:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Add infra-cloud spec https://review.openstack.org/186960 | 23:32 |
| jeblair | SpamapS: ^ let me know what you think. i lifted some stuff from the system-config change and added you as a co-author | 23:33 |
| *** gokrokve_ has joined #openstack-infra | 23:33 | |
| mordred | jeblair: how much do you want me to quibble over terminology in that spec? | 23:34 |
| jeblair | mordred: if it alters meaning or makes me look less stupid, i'm happy for you to. :) | 23:35 |
| *** mdrnstm has joined #openstack-infra | 23:37 | |
| *** mdrnstm has joined #openstack-infra | 23:37 | |
| *** gokrokve has quit IRC | 23:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Add infra-cloud spec https://review.openstack.org/186960 | 23:37 |
| jlvillal | Has there ever been discussion about setting up a 'manifest' git repository? So could setup manifest files and use 'repo'? | 23:37 |
| jlvillal | I was envisioning having an 'ironic' manifest and I do a 'repo init' and 'repo sync' and get all projects related to ironic. | 23:38 |
| mordred | jlvillal: we actually decided to not use repo early on | 23:38 |
| mordred | jlvillal: largely because it had a workflow more outside of the normal git workflow and we were concerned with keeping things as close to regular git for folks as we could | 23:39 |
| mordred | jlvillal: this is not to say that such a thing is necessarily a bad idea - just that we have not thought much about repo since then | 23:39 |
| jlvillal | mordred: To me it would be useful. Not to make it required but optional. | 23:39 |
| *** alexsyip has quit IRC | 23:40 | |
| jlvillal | I can easily pull down all projects related to some master project. | 23:40 |
| jeblair | i'm open to it... i think key requirements are: not required for contributions; doesn't interfere with the existing contribution workflow; scalable and generalizable to different projects | 23:41 |
| *** puranamr_ has joined #openstack-infra | 23:41 | |
| *** puranamr has joined #openstack-infra | 23:41 | |
| jeblair | but i haven't looked at it in a while, so i would benefit from a written proposal for how the manifests would be set up | 23:42 |
| jlvillal | jeblair: Thanks. I'll see if I get motivated to write up something :) | 23:42 |
| nibalizer | does that basically become a map of projects to related repos? | 23:44 |
| jlvillal | nibalizer: It can work like that. | 23:44 |
| *** puranamr_ has quit IRC | 23:44 | |
| *** puranamr has quit IRC | 23:44 | |
| clarkb | jeblair I wonder how that differs from zuul cloner other than lacking ability to set up zuul refs | 23:44 |
| jeblair | dstanek: i don't think there are any major gertty changes in flight right now, so if you have time to update your changes, we can probably get them in soon | 23:44 |
| jeblair | clarkb: that's a point -- could share clonemap files | 23:45 |
| *** geoffarnold has joined #openstack-infra | 23:45 | |
| *** baoli has quit IRC | 23:45 | |
| jeblair | jlvillal: http://docs.openstack.org/infra/zuul/cloner.html is the tool we're moving the ci system to use for setting up repos; clarkb has been making it friendlier for local use | 23:46 |
| jlvillal | jeblair: Thanks. I will look at it. | 23:46 |
| clarkb | the last missing piece for local use is easy install (right now you have to install all of zuul) | 23:46 |
| jeblair | clarkb: should we fork it into its own repo? | 23:47 |
| clarkb | mordred seemed to think we could make something like pip install zuul[cloner] work | 23:47 |
| clarkb | but splitting is another option | 23:47 |
| jeblair | i'm interested in exploring the first option first :) | 23:48 |
| mordred | I see no reason we can't | 23:48 |
| *** RichardRaseley has quit IRC | 23:49 | |
| *** spzala has quit IRC | 23:50 | |
| *** fawadkhaliq has joined #openstack-infra | 23:52 | |
| *** mdrnstm has quit IRC | 23:52 | |
| *** xyang1 has quit IRC | 23:52 | |
| geoffarnold | Newbie q: I've pushed the first of three related commits for a new Stackforge project, and the first includes the (shared) acls and channels. Must I wait until that one goes through, or can I stack the other two commits? | 23:52 |
| *** bswartz has joined #openstack-infra | 23:53 | |
| jeblair | geoffarnold: what are the other two? | 23:53 |
| *** mdrnstm has joined #openstack-infra | 23:53 | |
| jeblair | oh, for two other repos related to the same project? | 23:53 |
| *** mdrnstm is now known as Guest46643 | 23:53 | |
| geoffarnold | First is https://review.openstack.org/#/c/186906/3 | 23:53 |
| *** ivar-laz_ has quit IRC | 23:54 | |
| *** Guest46643 is now known as needscoffee | 23:54 | |
| *** needscoffee has joined #openstack-infra | 23:54 | |
| geoffarnold | Yes, it's a distributed system: one for the pub service, one for the subscriber service, one for the cli | 23:54 |
| *** ajmiller has quit IRC | 23:54 | |
| *** AzherKhan has joined #openstack-infra | 23:54 | |
| *** needscoffee is now known as mdrnstm | 23:54 | |
| *** ivar-lazzaro has joined #openstack-infra | 23:54 | |
| jeblair | geoffarnold: yeah, then i would stack them so that the other two git-depend on the first (or A-B-C), and then it's fine to go ahead and push them all up. | 23:55 |
| *** sdake_ has joined #openstack-infra | 23:56 | |
| *** gokrokve_ has quit IRC | 23:56 | |
| *** fawadkhaliq has quit IRC | 23:56 | |
| *** gokrokve has joined #openstack-infra | 23:57 | |
| geoffarnold | So add a Depends-On: <Gerrit Change-Id> to the commit for #2 and #3 (#2 depends on #1, #3 on #2). Got it - thanks | 23:57 |
| jeblair | geoffarnold: you don't need to use depends-on, you can just commit one after the other in your repo and run 'git review' to upload all three | 23:58 |
| geoffarnold | three repos, though - they're independently deployable | 23:58 |
| jeblair | geoffarnold: gerrit will recognize the fact that your last commit was based on the one before it, and it on the one before that, and will create three changes, but link them together in such a way that they will merge in the order you committed them. | 23:59 |
| geoffarnold | OK. Just learn to trust gerri.... ;-) | 23:59 |
| geoffarnold | gerrit | 23:59 |
| geoffarnold | Thanks | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!