| *** puranamr has quit IRC | 00:00 | |
| *** puranamr has joined #openstack-infra | 00:00 | |
| *** near89 has left #openstack-infra | 00:01 | |
| *** ybathia has quit IRC | 00:01 | |
| *** sridhar_ram has quit IRC | 00:01 | |
| *** dangers is now known as dangers_away | 00:02 | |
| clarkb | http://logs.openstack.org/54/211754/4/check/gate-dsvm-nodepool-nv/f5bb930/logs/devstacklog.txt.gz#_2015-08-17_23_33_20_041 neutron didn't start when testing nodepool | 00:04 |
|---|---|---|
| clarkb | lets see if that is consistent | 00:04 |
| fungi | does neutron often fail to start in devstack? | 00:05 |
| *** achanda_ has joined #openstack-infra | 00:05 | |
| clarkb | dougwig: http://logs.openstack.org/54/211754/4/check/gate-dsvm-nodepool-nv/f5bb930/logs/screen-q-svc.txt.gz#_2015-08-17_23_32_21_497 that appears to be the error is that all related to what we are doing with the advanced services cleanup in devstack/devstack-gate? | 00:05 |
| *** sergmelikyan has quit IRC | 00:06 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm publish to eslint-config-openstack https://review.openstack.org/199725 | 00:07 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Switched npm-run-test to npm-test https://review.openstack.org/212154 | 00:07 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm publish job. https://review.openstack.org/199715 | 00:07 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm tarball upload job https://review.openstack.org/212153 | 00:07 |
| *** achanda has quit IRC | 00:07 | |
| *** asettle has joined #openstack-infra | 00:08 | |
| *** asettle has quit IRC | 00:09 | |
| *** achanda_ has quit IRC | 00:09 | |
| *** asettle has joined #openstack-infra | 00:10 | |
| *** tongli has quit IRC | 00:10 | |
| *** dannywilson has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Drop these config switches as they are no longer being used https://review.openstack.org/199672 | 00:12 |
| openstackgerrit | Merged openstack-infra/project-config: Horizon linting should be voting https://review.openstack.org/213344 | 00:13 |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Fix parameters for jeepyb::openstackwatch https://review.openstack.org/208758 | 00:13 |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Add Gemfile and puppet 4 checks https://review.openstack.org/208759 | 00:13 |
| openstackgerrit | Merged openstack-infra/shade: Change functional testing to use clouds.yaml https://review.openstack.org/212083 | 00:13 |
| clarkb | fungi: I am not sure but its the vpn service giving us trouble and there were related things I think | 00:14 |
| *** sergmelikyan has joined #openstack-infra | 00:15 | |
| *** geoffarnold has quit IRC | 00:17 | |
| *** zz_dimtruck is now known as dimtruck | 00:18 | |
| *** abdelwas has quit IRC | 00:19 | |
| *** sabeen has joined #openstack-infra | 00:20 | |
| ianw | SergeyLukjanov: https://review.openstack.org/#/c/213535/ <- i don't think this is quite ready to go, the dependent change hasn't merged | 00:21 |
| SergeyLukjanov | ianw, thx for the notice, I've removed W+1 | 00:22 |
| *** ihrachyshka has quit IRC | 00:22 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/213942 | 00:22 |
| *** bharath has joined #openstack-infra | 00:23 | |
| *** sharikapongubala has quit IRC | 00:25 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Recommend --no-ff on merges from feature to master https://review.openstack.org/210124 | 00:25 |
| openstackgerrit | Merged openstack-infra/infra-manual: Insert further details of the procedure https://review.openstack.org/209753 | 00:25 |
| *** tqtran has quit IRC | 00:25 | |
| *** tsekiyama has quit IRC | 00:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding RALLY_SCENARIO env for Cue job https://review.openstack.org/201285 | 00:26 |
| *** puranamr has quit IRC | 00:26 | |
| *** ivar-laz_ has joined #openstack-infra | 00:26 | |
| asselin_ | submitted this neutron devstack bug earlier today: https://bugs.launchpad.net/neutron/+bug/1485718 | 00:27 |
| openstack | Launchpad bug 1485718 in neutron "Devstack neutron intermittently fails on neutron-debug probe-create --device-owner compute" [Undecided,New] | 00:27 |
| openstackgerrit | Merged openstack-infra/project-config: Add check-requirements job to networking-ofagent project https://review.openstack.org/213639 | 00:27 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Add config option for asyncLoggingBufferSize https://review.openstack.org/213848 | 00:27 |
| openstackgerrit | Merged openstack-infra/project-config: Create an os-testr venv for running subunit2html https://review.openstack.org/213238 | 00:27 |
| *** ivar-laz_ has quit IRC | 00:27 | |
| openstackgerrit | Merged openstack-infra/project-config: midonet: Update ML2 job https://review.openstack.org/211420 | 00:28 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Test fix for ironic ephemeral drive problem https://review.openstack.org/205273 | 00:28 |
| *** ivar-laz_ has joined #openstack-infra | 00:28 | |
| *** ashleighfarnham has quit IRC | 00:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Add python3-jobs for searchlight https://review.openstack.org/212103 | 00:28 |
| *** ivar-lazzaro has quit IRC | 00:29 | |
| openstackgerrit | Merged openstack-infra/project-config: remove keystone extensions from openstackclient functional job https://review.openstack.org/211701 | 00:29 |
| openstackgerrit | Merged openstack-infra/project-config: Add acls for stable/* branch to fuel-mirror project https://review.openstack.org/209223 | 00:29 |
| openstackgerrit | Merged openstack-infra/project-config: Remove rally-jobs job group https://review.openstack.org/209786 | 00:29 |
| *** woodster_ has quit IRC | 00:30 | |
| *** ivar-laz_ has quit IRC | 00:31 | |
| *** mtanino has quit IRC | 00:32 | |
| *** shashank_hegde has quit IRC | 00:32 | |
| SpamapS | crinkle: thanks for the pointer to the inventory json. Now what we have to do is move all of the ipmi passwords into hiera. :-P | 00:33 |
| *** puranamr has joined #openstack-infra | 00:33 | |
| asselin_ | clarkb, akraminsky another difference is that we have cloud credentials in nodepool.yaml. upstream uses clouds.yaml https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/templates/nodepool/clouds.yaml.erb | 00:33 |
| SpamapS | crinkle: which has reminded me of why I was thinking we might want our own local hiera for now | 00:33 |
| crinkle | SpamapS: did we decide we couldn't set them all to the same password? | 00:34 |
| *** davideagnello has quit IRC | 00:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable merge commits for fuel-plugin-influxdb-grafana-release group https://review.openstack.org/213627 | 00:34 |
| *** dims has joined #openstack-infra | 00:34 | |
| *** davideagnello has joined #openstack-infra | 00:34 | |
| SpamapS | crinkle: we can, but that is reliant on the HP NOC doing that work. | 00:34 |
| *** ZZelle-off has quit IRC | 00:34 | |
| SpamapS | crinkle: not something I want to wait for. :) | 00:34 |
| crinkle | aha | 00:35 |
| SpamapS | OR we can do it with an expect script | 00:35 |
| SpamapS | crinkle: is it exceedingly different to just populate a local hiera with 40 or so values? | 00:36 |
| SpamapS | difficult rather | 00:36 |
| nibalizer | shouldn't be | 00:36 |
| nibalizer | its just yaml | 00:36 |
| nibalizer | althought I don't have all the context | 00:36 |
| SpamapS | I just want to generate that yaml file for now so I can get the rest of inventory checked in. :) | 00:37 |
| *** puranamr has quit IRC | 00:37 | |
| SpamapS | so I'd have my own little local yaml that I could manually put on the box before running apply. | 00:37 |
| crinkle | SpamapS: not difficult, just will be a lot of parameters | 00:38 |
| *** davideagnello has quit IRC | 00:38 | |
| *** davideagnello has joined #openstack-infra | 00:38 | |
| SpamapS | crinkle: indeed.. but seems like something we have to prepare for. And I can do it programatically. | 00:38 |
| SpamapS | as in, I can take the full inventory as it is now, and mold it into one that can be checked in | 00:38 |
| *** tiswanso has joined #openstack-infra | 00:38 | |
| SpamapS | and at the same time it will spit out the hiera yaml needed. | 00:38 |
| SpamapS | ok I'm just going to do that. | 00:39 |
| crinkle | okie | 00:39 |
| *** akraminsky has quit IRC | 00:39 | |
| *** ashleighfarnham has joined #openstack-infra | 00:40 | |
| *** tiswanso has quit IRC | 00:40 | |
| *** tiswanso has joined #openstack-infra | 00:41 | |
| *** davideagnello has quit IRC | 00:41 | |
| *** davideag_ has joined #openstack-infra | 00:41 | |
| fungi | wow, gate-tripleo-ironic-overcloud-f21puppet-ceph has up to a 90-minute wait built into it | 00:42 |
| fungi | "Waiting for the overcloud stack to be ready" wait_for_stack_ready -w 5400 10 overcloud | 00:43 |
| *** mriedem has joined #openstack-infra | 00:43 | |
| fungi | the one i'm waiting to finish has spent almost an hour in that state so far | 00:43 |
| *** habib has quit IRC | 00:44 | |
| *** ashleighfarnham has quit IRC | 00:44 | |
| asselin_ | clarkb, this looks related: https://review.openstack.org/#/c/212428/ | 00:45 |
| *** mriedem1 has quit IRC | 00:45 | |
| *** crc32 has joined #openstack-infra | 00:46 | |
| *** sergmelikyan has quit IRC | 00:46 | |
| *** sputnik13 has quit IRC | 00:46 | |
| asselin_ | os-client-config==1.6.3 # git sha c422fe8 | 00:46 |
| *** baoli has joined #openstack-infra | 00:46 | |
| *** sergmelikyan has joined #openstack-infra | 00:48 | |
| *** mriedem has quit IRC | 00:48 | |
| *** sergmelikyan has quit IRC | 00:48 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:49 | |
| *** mriedem has joined #openstack-infra | 00:50 | |
| *** Sukhdev has quit IRC | 00:50 | |
| *** baoli has quit IRC | 00:51 | |
| *** moravec has joined #openstack-infra | 00:52 | |
| *** hdd has quit IRC | 00:53 | |
| dougwig | clarkb: yes, related. Why is vpn enabled? What job? | 00:54 |
| clarkb | douwig its the nodepool dsvm job and I dont think vpn is enabled but I did override the services list | 00:54 |
| clarkb | I should just disable advanced services? | 00:55 |
| krotscheck | I have a trusty node here that claims there's no python 2.7. Did that get evicted at some point in the past? | 00:55 |
| clarkb | krotscheck trusty is python3 by default so you have to ensure 2/7 is installed | 00:56 |
| dougwig | It still expects lb and fw, but vpn is out of the default list. Or I can post a workaround | 00:56 |
| dougwig | clarkb: ^^ | 00:56 |
| krotscheck | clarkb: Gotcha, thanks :) | 00:56 |
| *** moravec has quit IRC | 00:56 | |
| clarkb | dougwig we dobt need the service for nodepool so likely can just remove | 00:56 |
| dougwig | clarkb: I'm not familiar enough with nodepool images to be certain, but assuming it's a pre cache reinstall situation, that is correct. | 00:57 |
| dougwig | Pre install not reinstall. I'm on my phone, sorry. /) | 00:59 |
| *** jeadr has quit IRC | 01:01 | |
| *** jeadr has joined #openstack-infra | 01:02 | |
| *** sergmelikyan has joined #openstack-infra | 01:02 | |
| clarkb | dougwig this is a job that deploys nodepool against devstack, I can fix | 01:03 |
| *** sharikapongubala has joined #openstack-infra | 01:03 | |
| *** dimtruck is now known as zz_dimtruck | 01:03 | |
| krotscheck | clarkb: So, the 1.9.4 version of stankevitch/python, which seems to be the one currently in use in the bug I'm tracking down, does this: https://github.com/stankevich/puppet-python/blob/1.9.4/manifests/params.pp | 01:04 |
| krotscheck | Note the complete lack of 2.7. | 01:04 |
| *** sharikapongubala has quit IRC | 01:04 | |
| jeblair | krotscheck: what's this for? | 01:05 |
| krotscheck | jeblair: refstack: http://logs.openstack.org/69/198869/6/check/gate-infra-puppet-apply-bare-trusty/6185973/console.html#_2015-08-03_20_01_32_682 | 01:05 |
| *** thcipriani is now known as thcipriani|afk | 01:06 | |
| krotscheck | jeblair: This review in particular -> https://review.openstack.org/#/c/198869/ | 01:06 |
| jeblair | what's puppet-refstack using that module to do? | 01:06 |
| *** sergmelikyan has quit IRC | 01:07 | |
| krotscheck | jeblair: Install python -> http://git.openstack.org/cgit/openstack-infra/puppet-refstack/tree/manifests/api.pp#n30 | 01:07 |
| *** sharikapongubala has joined #openstack-infra | 01:07 | |
| jeblair | krotscheck: i don't think we use that module anywhere else and we run lots of python | 01:08 |
| fungi | seems it doesn't do that so well. i think based on the errors it emits that it | 01:08 |
| fungi | 's only meant to install pythons other than 2.7 | 01:08 |
| fungi | the modules we have which use that don't pass in a python_version class parameter | 01:08 |
| *** xyang1 has quit IRC | 01:08 | |
| krotscheck | Well, the one that was specified in the package itself is version 1.6.6, which works fine. | 01:08 |
| jeblair | fungi: what uses it? | 01:08 |
| fungi | puppet-storyboard i think was one of the examples i fished out | 01:08 |
| fungi | seems like there's at least one other though | 01:09 |
| krotscheck | In this case I was using it because it provided an idempotent way of creating a venv. | 01:09 |
| jeblair | krotscheck: i don't follow "Well, the one that was specified in the package itself is version 1.6.6, which works fine." ? | 01:09 |
| jeblair | krotscheck: ah | 01:09 |
| jeblair | krotscheck: we don't generally create venvs for servers | 01:09 |
| krotscheck | jeblair: http://git.openstack.org/cgit/openstack-infra/puppet-refstack/tree/metadata.json | 01:09 |
| jeblair | krotscheck: can we avoid doing that with refstack? i think it will vastly simplify things | 01:09 |
| krotscheck | Though that's >= | 01:10 |
| krotscheck | Hrm | 01:10 |
| krotscheck | Lovely how they break things for us. | 01:10 |
| krotscheck | jeblair: I'm sure we can. I'll circle back to the refstack team and let them know what changes are necessary | 01:10 |
| *** bpokorny_ has quit IRC | 01:10 | |
| jeblair | krotscheck: cool, thx. most (all?) of our python app servers just "pip install -U" a thing right into the system env | 01:11 |
| jeblair | (which usually works fine for single-purpose servers) | 01:11 |
| *** shashank_hegde has joined #openstack-infra | 01:12 | |
| *** rfolco has joined #openstack-infra | 01:12 | |
| *** thorst has joined #openstack-infra | 01:12 | |
| *** sharikapongubala has quit IRC | 01:12 | |
| *** mriedem has quit IRC | 01:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-cgit: WIP: haproxy statsd https://review.openstack.org/213952 | 01:14 |
| *** mtanino has joined #openstack-infra | 01:14 | |
| fungi | jenkins04, 05 and 07 have been cleaned up and restarted now. we're back at full compliment of jenkins masters again | 01:14 |
| jeblair | clarkb, fungi, pleia2: ^ that's 90% of the work to get haproxy stats into graphite. of course the last 10% is lots of fiddly config files and stuff. :) | 01:15 |
| *** Yanyanhu has joined #openstack-infra | 01:16 | |
| *** alexsyip has quit IRC | 01:17 | |
| *** sharikapongubala has joined #openstack-infra | 01:17 | |
| fungi | awesome | 01:17 |
| *** Yanyan has joined #openstack-infra | 01:17 | |
| *** sharikapongubala has quit IRC | 01:18 | |
| fungi | daemon that periodically reads from the statsd status info socket and spews it to a statsd server? | 01:19 |
| openstackgerrit | Merged openstack/requirements: Version bump for hacking https://review.openstack.org/213255 | 01:20 |
| *** hdd has joined #openstack-infra | 01:20 | |
| *** Yanyanhu has quit IRC | 01:21 | |
| fungi | yep, looks like a great solution | 01:21 |
| *** zhenguo has joined #openstack-infra | 01:21 | |
| *** pvaneck has quit IRC | 01:22 | |
| *** near89 has joined #openstack-infra | 01:23 | |
| openstackgerrit | Merged openstack/requirements: Add pyinotify to global-requirements.txt https://review.openstack.org/211975 | 01:23 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update title and chairs for tacker meeting https://review.openstack.org/213926 | 01:27 |
| *** salv-orlando has joined #openstack-infra | 01:29 | |
| *** near89 has left #openstack-infra | 01:29 | |
| *** sharikapongubala has joined #openstack-infra | 01:30 | |
| *** esker has joined #openstack-infra | 01:30 | |
| *** rfolco has quit IRC | 01:30 | |
| *** rick__ has joined #openstack-infra | 01:33 | |
| sthillma | Hi all, any chance I could get another core to take a look at/approve this change real quick: https://review.openstack.org/#/c/212221/ ? | 01:34 |
| *** ashleighfarnham has joined #openstack-infra | 01:34 | |
| *** salv-orlando has quit IRC | 01:35 | |
| *** esker has quit IRC | 01:36 | |
| *** rick__ has quit IRC | 01:37 | |
| *** rick-chen has joined #openstack-infra | 01:38 | |
| *** ashleighfarnham has quit IRC | 01:39 | |
| *** Swami has quit IRC | 01:39 | |
| *** zz_dimtruck is now known as dimtruck | 01:39 | |
| *** davideag_ has quit IRC | 01:41 | |
| *** sharikapongubala has quit IRC | 01:42 | |
| *** vadivelp has joined #openstack-infra | 01:43 | |
| vadivelp | hi | 01:43 |
| *** tjones has joined #openstack-infra | 01:43 | |
| vadivelp | i would like to add myself to the following group | 01:44 |
| vadivelp | https://review.openstack.org/#/admin/groups/1018,members | 01:44 |
| *** sharikapongubala has joined #openstack-infra | 01:44 | |
| vadivelp | can someone from the infra team can add me | 01:44 |
| *** tjones has left #openstack-infra | 01:44 | |
| *** pfallenop has quit IRC | 01:44 | |
| *** sthillma has quit IRC | 01:46 | |
| *** sharikapongubala has quit IRC | 01:46 | |
| *** sharikapongubala has joined #openstack-infra | 01:50 | |
| *** thorst has quit IRC | 01:50 | |
| *** yamahata has quit IRC | 01:51 | |
| *** sharikapongubala has quit IRC | 01:52 | |
| *** moravec has joined #openstack-infra | 01:53 | |
| *** angdraug has quit IRC | 01:58 | |
| *** moravec has quit IRC | 01:58 | |
| dougwig | clarkb, fungi: that error is because vpn flipped from implicit config to explicit, and you hit a corner case that wasn't patched to that effect. since the root gate issue was fixed in a different way, i've proposed a revert of that config change ( https://review.openstack.org/#/c/213963/ ), until it can be addressed while affecting fewer people. | 02:00 |
| *** sharikapongubala has joined #openstack-infra | 02:01 | |
| *** Kennan2 has joined #openstack-infra | 02:02 | |
| *** Kennan has quit IRC | 02:03 | |
| vadivelp | hi, may i know who is the PTL for #openstack-infra ? | 02:03 |
| vadivelp | i have a request, but not sure sure whom to make?.. | 02:03 |
| vadivelp | anybody can throw some light?... i want to add myself to the Gerrit group that represents the repository that i had created | 02:04 |
| *** yamamoto has joined #openstack-infra | 02:05 | |
| *** sharikapongubala has quit IRC | 02:06 | |
| *** Swami has joined #openstack-infra | 02:10 | |
| *** mtanino has quit IRC | 02:11 | |
| *** olaph has joined #openstack-infra | 02:13 | |
| *** yamamoto has quit IRC | 02:14 | |
| *** Swami has quit IRC | 02:16 | |
| *** w_ has joined #openstack-infra | 02:21 | |
| *** achanda has joined #openstack-infra | 02:23 | |
| *** olaph has quit IRC | 02:23 | |
| pleia2 | vadivelp: what is the repository? I can get you added | 02:24 |
| pleia2 | aha, I see, networking-ale-omniswitch | 02:27 |
| *** jerryz has quit IRC | 02:28 | |
| pleia2 | vadivelp: you've been added to the networking-ale-omniswitch-core group, you can add others as needed | 02:28 |
| *** ashleighfarnham has joined #openstack-infra | 02:28 | |
| *** dims has quit IRC | 02:30 | |
| *** yamahata has joined #openstack-infra | 02:30 | |
| *** yamahata has quit IRC | 02:32 | |
| *** yamahata has joined #openstack-infra | 02:32 | |
| *** ashleighfarnham has quit IRC | 02:33 | |
| *** bpokorny has joined #openstack-infra | 02:35 | |
| *** jeadr has quit IRC | 02:38 | |
| *** jeadr has joined #openstack-infra | 02:39 | |
| *** pcrews has joined #openstack-infra | 02:39 | |
| *** oomichi has joined #openstack-infra | 02:40 | |
| *** bswartz has joined #openstack-infra | 02:42 | |
| *** esker has joined #openstack-infra | 02:51 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add controller class for Infra Cloud https://review.openstack.org/209698 | 02:53 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add compute class for Infra Cloud https://review.openstack.org/213980 | 02:53 |
| *** jeadr has quit IRC | 02:59 | |
| *** jeadr has joined #openstack-infra | 03:00 | |
| *** esker has quit IRC | 03:02 | |
| *** esker has joined #openstack-infra | 03:04 | |
| openstackgerrit | Merged openstack/requirements: Bump tooz to version 1.19.0 https://review.openstack.org/205730 | 03:04 |
| *** bpokorny has quit IRC | 03:06 | |
| *** yamamoto has joined #openstack-infra | 03:06 | |
| *** unicell1 has quit IRC | 03:13 | |
| *** shashank_hegde has quit IRC | 03:13 | |
| *** salv-orlando has joined #openstack-infra | 03:19 | |
| *** PinkFreud has quit IRC | 03:21 | |
| *** bpokorny has joined #openstack-infra | 03:23 | |
| *** camunoz has quit IRC | 03:24 | |
| *** otter768 has quit IRC | 03:24 | |
| *** esker has quit IRC | 03:26 | |
| *** salv-orlando has quit IRC | 03:26 | |
| *** esker has joined #openstack-infra | 03:26 | |
| *** sharikapongubala has joined #openstack-infra | 03:29 | |
| *** changbl has quit IRC | 03:30 | |
| *** pvaneck has joined #openstack-infra | 03:31 | |
| *** crc32 has quit IRC | 03:31 | |
| *** sdake has joined #openstack-infra | 03:32 | |
| *** shashank_hegde has joined #openstack-infra | 03:34 | |
| *** tiswanso has quit IRC | 03:35 | |
| *** sdake_ has quit IRC | 03:36 | |
| *** camunoz has joined #openstack-infra | 03:36 | |
| *** esker has quit IRC | 03:37 | |
| *** esker has joined #openstack-infra | 03:38 | |
| asselin | clarkb, I think there's something with how the request is built. I forced debug mode and tried the curl output directly on different providers and got the same error message. | 03:38 |
| asselin | http://paste.openstack.org/show/419928/ | 03:39 |
| *** unicell has joined #openstack-infra | 03:40 | |
| *** sabeen has quit IRC | 03:41 | |
| asselin | paste corrected: http://paste.openstack.org/show/419929/ | 03:41 |
| *** markvoelker has quit IRC | 03:42 | |
| *** sdake_ has joined #openstack-infra | 03:42 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Kolla: Add build gate for ubuntu binary https://review.openstack.org/213530 | 03:43 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Kolla: Add build gate for ubuntu source https://review.openstack.org/213531 | 03:43 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Kolla: add a deprecated vs templated build model https://review.openstack.org/213535 | 03:43 |
| *** jeadr has quit IRC | 03:45 | |
| *** sdake has quit IRC | 03:45 | |
| *** jeadr has joined #openstack-infra | 03:45 | |
| *** fedexo has joined #openstack-infra | 03:47 | |
| *** alivigni has joined #openstack-infra | 03:58 | |
| *** alivigni has quit IRC | 03:58 | |
| *** esker has quit IRC | 03:58 | |
| *** esker has joined #openstack-infra | 04:00 | |
| *** dimtruck is now known as zz_dimtruck | 04:00 | |
| *** zhenguo has quit IRC | 04:06 | |
| *** gildub has quit IRC | 04:06 | |
| *** shashank_hegde has quit IRC | 04:09 | |
| *** coolsvap|away is now known as coolsvap | 04:10 | |
| *** shashank_hegde has joined #openstack-infra | 04:12 | |
| *** esker has quit IRC | 04:13 | |
| *** esker has joined #openstack-infra | 04:13 | |
| *** tjones has joined #openstack-infra | 04:14 | |
| *** tjones has left #openstack-infra | 04:14 | |
| *** esker has quit IRC | 04:14 | |
| *** esker has joined #openstack-infra | 04:14 | |
| *** ashleighfarnham has joined #openstack-infra | 04:18 | |
| *** flepied has joined #openstack-infra | 04:18 | |
| *** esker has quit IRC | 04:20 | |
| *** sdake_ is now known as sdake | 04:22 | |
| *** fedexo has quit IRC | 04:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove extra pyhon-jobs from networking-ofagent https://review.openstack.org/213656 | 04:22 |
| *** ashleighfarnham has quit IRC | 04:22 | |
| *** gmann has quit IRC | 04:23 | |
| *** armax has joined #openstack-infra | 04:23 | |
| *** ayoung has quit IRC | 04:25 | |
| Nakato | lifeless: WRT making constraints opt in without requiring new jobs (pyNN-constraints), opt in could look like this. | 04:26 |
| Nakato | Add a check for upper-constraints.txt into run-tox.sh, then export UPPER_CONSTRAINTS_ARGUMENT as '-c /path/to/upper-constraints.txt', and if the install_command has added {env:UPPER_CONSTRAINTS_ARGUMENT:} it'll use that in gate, not effecting dev workflow, but allowing to opt in by exporting UPPER_CONSTRAINTS_ARGUMENT. | 04:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup some bash comparisions https://review.openstack.org/212663 | 04:28 |
| lifeless | Nakato: thats still a change in the project testing interface though | 04:29 |
| lifeless | Nakato: clark's request can be read backwards: it should be possible for someone prepping a tree to work with openstack's gate to predict how to drive it | 04:29 |
| lifeless | Nakato: now, it may be that the PTI is underdocumented today (in that there are a billion variables we don't document in the governance docs) | 04:29 |
| lifeless | or it may be that it over-specifies (in that we don't need to specify what we do today) | 04:30 |
| lifeless | or the balance might be right | 04:30 |
| Nakato | lifeless: Okay, I'm confused then, how do we implment it without making any change. | 04:30 |
| lifeless | Nakato: sorry, I may have tied two tings together inappropriately | 04:30 |
| lifeless | Nakato: were you asking in the context of the governance docs proposal | 04:31 |
| lifeless | Nakato: or the local dev experience discussion ? | 04:31 |
| lifeless | Nakato: [or something else :)] | 04:31 |
| *** PinkFreud has joined #openstack-infra | 04:32 | |
| *** kiran-r has joined #openstack-infra | 04:32 | |
| Nakato | So reading before governence there was conversation about how to add it in with minimal changes. adding '-constraints' didn't seem liked, nor an empty constraints file with comments. So the above deals with both things, making it opt in by projects and developers without having to run pyNN-constraints | 04:33 |
| lifeless | ok | 04:33 |
| Nakato | I'm assuming that needs to be flushed out before I can go do anything to goverence. | 04:33 |
| lifeless | so -constraints is liked | 04:33 |
| lifeless | it just is something the infra folk feel should be added to the project testing interface | 04:33 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup some bash comparisions https://review.openstack.org/212663 | 04:33 |
| lifeless | I like it because its clear - you can look at the job name and know whether this job is subject to vaguaries or is locked down to be reliable | 04:34 |
| lifeless | s/job/tox name/ but both really | 04:34 |
| Nakato | So it seems I'm missing what is the project testing interface is. | 04:34 |
| lifeless | governance.openstack.org/reference/project-testing-interface.html | 04:35 |
| Nakato | Okay, so the goverenence docks. | 04:35 |
| lifeless | with an http because chromium is stupid | 04:35 |
| *** deepakcs has joined #openstack-infra | 04:35 | |
| lifeless | that specific sub doc, yes | 04:35 |
| Nakato | lifeless: Right, I'll get to work on adding that into there. That clarifies things immensely, thanks! | 04:36 |
| deepakcs | hi all, gerrit still acting strnage for me. I am trying to push a new patchset with new changes , but git review returns "rejected: no new changes" as the error | 04:36 |
| lifeless | yw | 04:36 |
| lifeless | Nakato: one specific that occurs to me | 04:36 |
| deepakcs | has anyone else faced same issue ? Should i just abandon my existing patch and send a new one w/ new change ID ? | 04:36 |
| lifeless | Nakato: you probably want to specify that a project MUST implement py27 and MAY implement py27-constraints | 04:36 |
| deepakcs | lifeless: jhesketh ^^ | 04:37 |
| lifeless | Nakato: or something like that | 04:37 |
| *** kiran-r has quit IRC | 04:37 | |
| lifeless | deepakcs: that means you have a change with a new hash but no body or parent changes | 04:37 |
| lifeless | deepakcs: e.g. you did a no-op rebase that resulted in a new hash or something | 04:37 |
| lifeless | deepakcs: change a char somewhere. | 04:37 |
| *** hdd has quit IRC | 04:37 | |
| deepakcs | lifeless: ok will try, thanks | 04:37 |
| deepakcs | it has a change, since i added a tst case | 04:38 |
| deepakcs | so not sure why it says no change! | 04:38 |
| lifeless | is it in a stack of changes? | 04:38 |
| deepakcs | lifeless: ^^ | 04:38 |
| deepakcs | yes, | 04:38 |
| lifeless | one of your lower changes is the problem | 04:38 |
| jhesketh | deepakcs: which change? | 04:38 |
| deepakcs | jhesketh: lifeless https://review.openstack.org/#/c/212518/ | 04:38 |
| *** shashank_hegde has quit IRC | 04:38 | |
| deepakcs | jhesketh: lifeless i am trying to send patch set 2, which has new changes (test_driver.py), but it says no change to me (by git review) | 04:39 |
| *** Sree has joined #openstack-infra | 04:39 | |
| lifeless | Nakato: ok, I'm offlining to finish this talk before I give it in 2h | 04:39 |
| Nakato | lifeless: Roger, thanks for your time. | 04:39 |
| *** flepied has quit IRC | 04:40 | |
| jhesketh | deepakcs: what sha1 does git show have for the repo where you're running git review? | 04:40 |
| deepakcs | jhesketh: d7427576d5a2c388b0a53c5a3fafff5c6c86b1c7 | 04:42 |
| *** markvoelker has joined #openstack-infra | 04:42 | |
| jhesketh | deepakcs: mind doing a git review -vvvv into a pastebin? | 04:44 |
| deepakcs | jhesketh: sure | 04:44 |
| jhesketh | deepakcs: while there, a copy of git show would also be handy | 04:46 |
| deepakcs | jhesketh: http://paste.fedoraproject.org/256107/14398731 | 04:46 |
| deepakcs | jhesketh: it has it ` git show --stat` | 04:46 |
| jhesketh | perfect, thanks | 04:46 |
| deepakcs | jhesketh: let me know what u find :) | 04:47 |
| *** markvoelker has quit IRC | 04:47 | |
| *** jeadr has quit IRC | 04:49 | |
| jhesketh | deepakcs: hmm, nothing obvious | 04:50 |
| *** jeadr has joined #openstack-infra | 04:50 | |
| jhesketh | deepakcs: did you try to submit this yesterday? | 04:50 |
| deepakcs | jhesketh: :) strange right | 04:50 |
| deepakcs | jhesketh: yes.. it tried twice. it hung, once i did ctrl-c too | 04:50 |
| deepakcs | jhesketh: guess its in bad shape after that | 04:51 |
| deepakcs | jhesketh: but if gerrit was rebooted, it should have any past context, so not sure why its not working | 04:51 |
| jhesketh | deepakcs: right, that's likely it | 04:51 |
| jhesketh | it's probably in an odd state | 04:51 |
| deepakcs | jhesketh: so whats the deal, abandon and send patch with new change ID ? | 04:52 |
| jhesketh | rather than poking around trying to fix gerrit, it's probably easier (and hopefully less likely to fail) to do just that | 04:52 |
| *** bpokorny has quit IRC | 04:52 | |
| deepakcs | jhesketh: ok will do, thanks | 04:52 |
| jhesketh | deepakcs: another alternative is to touch the change again and try | 04:52 |
| deepakcs | jhesketh: just did a git rebaste master and sha1 changed, so let me try now | 04:52 |
| deepakcs | jhesketh: worked ! | 04:53 |
| *** flepied has joined #openstack-infra | 04:53 | |
| jhesketh | deepakcs: interestingly it appeared as ps3 | 04:53 |
| deepakcs | jhesketh: i was hoping many would be in the same boat as I am , as many would have tried yesterday | 04:54 |
| deepakcs | jhesketh: lol, yeah just saw that! and #2 is missing | 04:54 |
| jhesketh | deepakcs: so the previous change/sha was likely in gerrit's index but the change wasn't in the database (or something, I am just guessing) | 04:54 |
| deepakcs | jhesketh: so that gerrit isn't transactional ;-) | 04:54 |
| jhesketh | heh, I'm hoping not many are in the same boat.. they shouldn't be though as gerrit was still working, just very slow | 04:54 |
| *** mestery has joined #openstack-infra | 04:59 | |
| *** tjones has joined #openstack-infra | 05:00 | |
| *** gyee has quit IRC | 05:04 | |
| *** mestery has quit IRC | 05:09 | |
| *** gmann has joined #openstack-infra | 05:11 | |
| *** ashleighfarnham has joined #openstack-infra | 05:12 | |
| *** mrmartin has joined #openstack-infra | 05:15 | |
| *** tjones has quit IRC | 05:15 | |
| *** ashleighfarnham has quit IRC | 05:16 | |
| *** pvaneck has quit IRC | 05:16 | |
| *** dims has joined #openstack-infra | 05:20 | |
| *** PinkFreud has quit IRC | 05:24 | |
| *** otter768 has joined #openstack-infra | 05:25 | |
| *** bharath has quit IRC | 05:26 | |
| *** dims has quit IRC | 05:26 | |
| *** otter768 has quit IRC | 05:30 | |
| *** jeadr has quit IRC | 05:30 | |
| *** jeadr has joined #openstack-infra | 05:31 | |
| *** yamamoto has quit IRC | 05:32 | |
| *** rlandy has quit IRC | 05:35 | |
| *** salv-orlando has joined #openstack-infra | 05:37 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Add a docs for storlets https://review.openstack.org/214006 | 05:38 |
| *** nadya has joined #openstack-infra | 05:40 | |
| *** ildikov has quit IRC | 05:40 | |
| *** camunoz has quit IRC | 05:41 | |
| *** camunoz has joined #openstack-infra | 05:41 | |
| *** nadya has quit IRC | 05:42 | |
| *** achanda has quit IRC | 05:43 | |
| *** shashank_hegde has joined #openstack-infra | 05:44 | |
| *** achanda has joined #openstack-infra | 05:47 | |
| *** salv-orlando has quit IRC | 05:50 | |
| *** Sukhdev has joined #openstack-infra | 05:51 | |
| *** yamamoto has joined #openstack-infra | 05:51 | |
| *** nadya has joined #openstack-infra | 05:51 | |
| *** pfallenop has joined #openstack-infra | 05:52 | |
| *** yamamoto has quit IRC | 05:53 | |
| *** yamamoto has joined #openstack-infra | 05:53 | |
| *** bharath has joined #openstack-infra | 05:55 | |
| *** yamamoto has quit IRC | 05:58 | |
| *** nadya has quit IRC | 06:02 | |
| *** moravec has joined #openstack-infra | 06:02 | |
| *** sergmelikyan has joined #openstack-infra | 06:03 | |
| *** serg_melikyan has joined #openstack-infra | 06:05 | |
| *** sergmelikyan has quit IRC | 06:05 | |
| *** yamamoto has joined #openstack-infra | 06:06 | |
| *** ashleighfarnham has joined #openstack-infra | 06:06 | |
| *** moravec has quit IRC | 06:06 | |
| *** yamamoto has quit IRC | 06:09 | |
| *** yamamoto has joined #openstack-infra | 06:09 | |
| *** juggler has quit IRC | 06:10 | |
| *** ashleighfarnham has quit IRC | 06:11 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Fix variables in nodepool.default.erb template https://review.openstack.org/211365 | 06:14 |
| *** BharatK has quit IRC | 06:14 | |
| *** hdd has joined #openstack-infra | 06:16 | |
| *** oomichi has quit IRC | 06:16 | |
| *** k4n0 has joined #openstack-infra | 06:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/214019 | 06:17 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Fix nodepool class parameter defaults https://review.openstack.org/211366 | 06:19 |
| *** salv-orlando has joined #openstack-infra | 06:21 | |
| *** coolsvap is now known as coolsvap|away | 06:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/213606 | 06:24 |
| *** vlaza has joined #openstack-infra | 06:24 | |
| *** zhenguo has joined #openstack-infra | 06:24 | |
| *** gildub has joined #openstack-infra | 06:26 | |
| *** pfallenop has quit IRC | 06:28 | |
| *** pfallenop has joined #openstack-infra | 06:28 | |
| *** pfallenop has joined #openstack-infra | 06:28 | |
| *** jtomasek has joined #openstack-infra | 06:29 | |
| *** stevemar has quit IRC | 06:29 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Separate the fetch remotes log dir and file https://review.openstack.org/192187 | 06:32 |
| *** zhenguo has quit IRC | 06:32 | |
| *** shashank_hegde has quit IRC | 06:34 | |
| *** sthillma has joined #openstack-infra | 06:34 | |
| *** nadya has joined #openstack-infra | 06:35 | |
| *** ybathia has joined #openstack-infra | 06:36 | |
| *** hdd has quit IRC | 06:37 | |
| *** sthillma_ has joined #openstack-infra | 06:37 | |
| *** asettle has quit IRC | 06:38 | |
| *** sthillma has quit IRC | 06:39 | |
| *** sthillma_ is now known as sthillma | 06:39 | |
| *** rm_work is now known as rm_work|away | 06:42 | |
| *** openstackgerrit_ has joined #openstack-infra | 06:43 | |
| *** markvoelker has joined #openstack-infra | 06:43 | |
| *** markvoelker has quit IRC | 06:48 | |
| *** BharatK has joined #openstack-infra | 06:50 | |
| *** shashank_hegde has joined #openstack-infra | 06:51 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Separate the fetch remotes log dir and file https://review.openstack.org/192187 | 06:51 |
| *** BharatK has quit IRC | 06:58 | |
| *** ashleighfarnham has joined #openstack-infra | 07:00 | |
| *** mrunge has joined #openstack-infra | 07:01 | |
| *** amotoki has quit IRC | 07:01 | |
| *** camunoz has quit IRC | 07:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/214019 | 07:02 |
| *** armax has quit IRC | 07:04 | |
| *** fawadkhaliq has joined #openstack-infra | 07:05 | |
| *** ashleighfarnham has quit IRC | 07:05 | |
| *** Sukhdev has quit IRC | 07:07 | |
| ttx | SpamapS: see https://wiki.openstack.org/wiki/Phabricator | 07:07 |
| *** ildikov has joined #openstack-infra | 07:07 | |
| ttx | Hmm, looks like mordred's test instance at http://15.126.194.141 is now dead -- his HP Cloud account probably just got wiped | 07:08 |
| *** ifarkas has joined #openstack-infra | 07:09 | |
| *** jyuso1 has joined #openstack-infra | 07:10 | |
| *** peristeri has joined #openstack-infra | 07:11 | |
| *** BharatK has joined #openstack-infra | 07:12 | |
| *** dguerri` is now known as dguerri | 07:13 | |
| *** rm_work|away is now known as rm_work | 07:14 | |
| *** markus_z has joined #openstack-infra | 07:14 | |
| *** camunoz has joined #openstack-infra | 07:14 | |
| *** sthillma has quit IRC | 07:16 | |
| *** dguerri is now known as dguerri` | 07:16 | |
| *** abregman has joined #openstack-infra | 07:17 | |
| *** jyuso1 has quit IRC | 07:17 | |
| *** scheuran has joined #openstack-infra | 07:18 | |
| *** vernhart has joined #openstack-infra | 07:20 | |
| *** vahidh has quit IRC | 07:21 | |
| *** fedexo has joined #openstack-infra | 07:22 | |
| *** dguerri` is now known as dguerri | 07:24 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Kolla: Add build gate for ubuntu source https://review.openstack.org/213531 | 07:25 |
| *** otter768 has joined #openstack-infra | 07:26 | |
| *** bharath has quit IRC | 07:30 | |
| *** stevemar has joined #openstack-infra | 07:30 | |
| *** otter768 has quit IRC | 07:31 | |
| *** dguerri is now known as dguerri` | 07:32 | |
| *** dguerri` is now known as dguerri | 07:32 | |
| *** achanda has quit IRC | 07:33 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Separate the fetch remotes log dir and file https://review.openstack.org/192187 | 07:33 |
| *** stevemar has quit IRC | 07:34 | |
| *** shardy has joined #openstack-infra | 07:36 | |
| *** camunoz has quit IRC | 07:36 | |
| *** fedexo has quit IRC | 07:38 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/puppet-zanata: Switch to a class require for deployment https://review.openstack.org/214053 | 07:39 |
| *** dguerri is now known as dguerri` | 07:39 | |
| *** ashleighfarnham has joined #openstack-infra | 07:41 | |
| *** amotoki has joined #openstack-infra | 07:42 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/tripleo-ci: Adding more CPU in the HA scenario https://review.openstack.org/213885 | 07:43 |
| openstackgerrit | Merged openstack-infra/nodepool: Run geard for nodepool tests https://review.openstack.org/194928 | 07:43 |
| *** ashleighfarnham has quit IRC | 07:46 | |
| *** dingyichen has quit IRC | 07:46 | |
| *** dingyichen has joined #openstack-infra | 07:47 | |
| *** dims has joined #openstack-infra | 07:48 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Separate the fetch remotes log dir and file https://review.openstack.org/192187 | 07:50 |
| *** arxcruz has joined #openstack-infra | 07:51 | |
| *** dims has quit IRC | 07:53 | |
| *** bradjones has quit IRC | 07:53 | |
| *** jistr has joined #openstack-infra | 07:55 | |
| *** bradjones has joined #openstack-infra | 07:56 | |
| *** bradjones has quit IRC | 07:56 | |
| *** bradjones has joined #openstack-infra | 07:56 | |
| *** jcoufal has joined #openstack-infra | 07:56 | |
| openstackgerrit | Tushar Gohad proposed openstack/requirements: Restrict PyECLib version to 1.0.7 https://review.openstack.org/214059 | 07:56 |
| *** devvesa has joined #openstack-infra | 07:56 | |
| *** fhubik has joined #openstack-infra | 07:57 | |
| *** fawadkhaliq has quit IRC | 08:01 | |
| *** fawadkhaliq has joined #openstack-infra | 08:01 | |
| *** ybathia has quit IRC | 08:03 | |
| openstackgerrit | Tushar Gohad proposed openstack/requirements: Restrict PyECLib version to 1.0.7 https://review.openstack.org/214059 | 08:04 |
| *** jeadr has quit IRC | 08:06 | |
| *** jeadr has joined #openstack-infra | 08:07 | |
| openstackgerrit | Tushar Gohad proposed openstack/requirements: Restrict PyECLib version to 1.0.7 https://review.openstack.org/214059 | 08:07 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 08:11 |
| *** derekh has joined #openstack-infra | 08:12 | |
| *** derekh has quit IRC | 08:12 | |
| openstackgerrit | Bharat Kumar Kobagana (BharatK) proposed openstack-infra/project-config: Renaming & moving Manila-GlusterFS jobs to check pipeline https://review.openstack.org/214065 | 08:12 |
| *** derekh has joined #openstack-infra | 08:13 | |
| *** moravec has joined #openstack-infra | 08:15 | |
| *** scheuran has quit IRC | 08:15 | |
| *** jeadr has quit IRC | 08:17 | |
| *** jeadr has joined #openstack-infra | 08:18 | |
| *** moravec has quit IRC | 08:19 | |
| *** scheuran has joined #openstack-infra | 08:19 | |
| *** shashank_hegde has quit IRC | 08:20 | |
| *** bharath_ has joined #openstack-infra | 08:30 | |
| *** HeOS has quit IRC | 08:32 | |
| openstackgerrit | Bharat Kumar Kobagana (BharatK) proposed openstack-infra/project-config: Renaming & moving Manila-GlusterFS jobs to check pipeline https://review.openstack.org/214065 | 08:33 |
| *** bharath_ has quit IRC | 08:35 | |
| *** ashleighfarnham has joined #openstack-infra | 08:35 | |
| *** jeadr has quit IRC | 08:39 | |
| *** ashleighfarnham has quit IRC | 08:40 | |
| *** jeadr has joined #openstack-infra | 08:40 | |
| *** ihrachyshka has joined #openstack-infra | 08:42 | |
| *** zeih has joined #openstack-infra | 08:42 | |
| *** markvoelker has joined #openstack-infra | 08:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove extra pyhon-jobs from networking-ofagent https://review.openstack.org/213656 | 08:45 |
| *** sshnaidm has joined #openstack-infra | 08:46 | |
| openstackgerrit | Jiri Stransky proposed openstack/diskimage-builder: Fedora: install older corosync https://review.openstack.org/213736 | 08:46 |
| *** markvoelker has quit IRC | 08:50 | |
| *** yamahata has quit IRC | 08:51 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Fedora: install older corosync https://review.openstack.org/213737 | 08:51 |
| *** dingyichen has quit IRC | 08:53 | |
| *** fhubik is now known as fhubik_brb | 08:55 | |
| *** fhubik_brb is now known as fhubik | 08:56 | |
| *** fhubik is now known as fhubik_brb | 08:58 | |
| *** sharikapongubala has quit IRC | 09:07 | |
| *** pelix has joined #openstack-infra | 09:09 | |
| *** fhubik_brb is now known as fhubik | 09:17 | |
| *** HeOS has joined #openstack-infra | 09:19 | |
| *** ihrachyshka has quit IRC | 09:19 | |
| openstackgerrit | Jiri Stransky proposed openstack/diskimage-builder: Fedora: install older corosync https://review.openstack.org/213736 | 09:19 |
| *** jyuso1 has joined #openstack-infra | 09:20 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Fedora: install older corosync https://review.openstack.org/213737 | 09:20 |
| *** otter768 has joined #openstack-infra | 09:27 | |
| *** fawadkhaliq has quit IRC | 09:30 | |
| *** stevemar has joined #openstack-infra | 09:31 | |
| *** otter768 has quit IRC | 09:32 | |
| *** ihrachyshka has joined #openstack-infra | 09:32 | |
| *** stevemar has quit IRC | 09:35 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: WIP: Fix the test_backend.sh scripts to work usefully CI https://review.openstack.org/213732 | 09:35 |
| *** afazkas has joined #openstack-infra | 09:42 | |
| *** jeadr has quit IRC | 09:44 | |
| *** jeadr has joined #openstack-infra | 09:45 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul: Remove unnecessary shebangs as managed by some entry_points https://review.openstack.org/214100 | 09:55 |
| *** fhubik is now known as fhubik_brb | 09:56 | |
| *** Yanyan has quit IRC | 09:56 | |
| *** madong_ is now known as larainema | 09:57 | |
| *** fhubik_brb is now known as fhubik | 10:01 | |
| *** zeih has quit IRC | 10:04 | |
| *** fhubik has quit IRC | 10:05 | |
| *** fhubik has joined #openstack-infra | 10:06 | |
| *** fhubik_ has joined #openstack-infra | 10:06 | |
| *** rick-chen has quit IRC | 10:08 | |
| *** jyuso1 has quit IRC | 10:13 | |
| *** sdake has quit IRC | 10:15 | |
| *** fhubik is now known as fhubik_brb | 10:16 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Pin heat and neutron https://review.openstack.org/214112 | 10:19 |
| *** bharath_ has joined #openstack-infra | 10:20 | |
| *** bharath_ has quit IRC | 10:24 | |
| *** ashleighfarnham has joined #openstack-infra | 10:25 | |
| *** moravec has joined #openstack-infra | 10:27 | |
| *** pc_m has joined #openstack-infra | 10:29 | |
| *** ashleighfarnham has quit IRC | 10:29 | |
| *** moravec has quit IRC | 10:31 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: WIP: Fix the test_backend.sh scripts to work usefully CI https://review.openstack.org/213732 | 10:33 |
| *** kiran-r has joined #openstack-infra | 10:36 | |
| *** claudiub has joined #openstack-infra | 10:38 | |
| *** Sree has quit IRC | 10:44 | |
| *** markvoelker has joined #openstack-infra | 10:46 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Bump cryptography version for Nova Bug 1481084 https://review.openstack.org/214136 | 10:48 |
| openstack | bug 1481084 in OpenStack Compute (nova) "Keypair creation fails when ssh public key comment contains spaces" [Undecided,In progress] https://launchpad.net/bugs/1481084 - Assigned to Davanum Srinivas (DIMS) (dims-v) | 10:48 |
| *** Sree has joined #openstack-infra | 10:48 | |
| *** markvoelker has quit IRC | 10:50 | |
| *** dims has joined #openstack-infra | 10:51 | |
| *** notnownikki has joined #openstack-infra | 10:53 | |
| *** PinkFreud has joined #openstack-infra | 10:55 | |
| *** salv-orlando has quit IRC | 10:57 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Pin heat and neutron https://review.openstack.org/214112 | 10:59 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Try alternate usernames to get logs from overcloud https://review.openstack.org/214139 | 11:01 |
| *** sc68cal has quit IRC | 11:02 | |
| *** sc68cal has joined #openstack-infra | 11:06 | |
| *** sc68cal has quit IRC | 11:06 | |
| *** sc68cal has joined #openstack-infra | 11:06 | |
| *** sc68cal has quit IRC | 11:07 | |
| *** _kiran_ has joined #openstack-infra | 11:07 | |
| *** k4n0 has quit IRC | 11:10 | |
| *** kiran-r has quit IRC | 11:11 | |
| *** _kiran_ has quit IRC | 11:12 | |
| *** yolanda has quit IRC | 11:12 | |
| *** yolanda has joined #openstack-infra | 11:12 | |
| *** yamamoto has quit IRC | 11:13 | |
| *** sc68cal has joined #openstack-infra | 11:16 | |
| *** sc68cal has quit IRC | 11:16 | |
| *** sc68cal has joined #openstack-infra | 11:16 | |
| *** baoli has joined #openstack-infra | 11:17 | |
| *** ashleighfarnham has joined #openstack-infra | 11:19 | |
| *** salv-orlando has joined #openstack-infra | 11:19 | |
| *** esker has joined #openstack-infra | 11:20 | |
| *** baoli has quit IRC | 11:21 | |
| *** ashleighfarnham has quit IRC | 11:24 | |
| *** zz_dimtruck is now known as dimtruck | 11:26 | |
| *** marcusvrn has joined #openstack-infra | 11:27 | |
| *** otter768 has joined #openstack-infra | 11:28 | |
| *** dims_ has joined #openstack-infra | 11:28 | |
| *** warpc__ has joined #openstack-infra | 11:29 | |
| *** dims has quit IRC | 11:30 | |
| *** stevemar has joined #openstack-infra | 11:32 | |
| *** otter768 has quit IRC | 11:33 | |
| *** dguitarbite has quit IRC | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** stevemar has quit IRC | 11:35 | |
| *** fawadkhaliq has joined #openstack-infra | 11:37 | |
| *** DuncanT has quit IRC | 11:38 | |
| *** dguitarbite has joined #openstack-infra | 11:38 | |
| *** marcusvrn has quit IRC | 11:38 | |
| *** tchaypo has quit IRC | 11:39 | |
| *** thorst has joined #openstack-infra | 11:40 | |
| *** ramishra has quit IRC | 11:40 | |
| *** gordc has joined #openstack-infra | 11:45 | |
| *** markvoelker has joined #openstack-infra | 11:46 | |
| *** fawadk has joined #openstack-infra | 11:46 | |
| *** yamamoto has joined #openstack-infra | 11:47 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** yamamoto has joined #openstack-infra | 11:47 | |
| *** fawadkhaliq has quit IRC | 11:47 | |
| *** yamamoto_ has joined #openstack-infra | 11:48 | |
| *** esker has quit IRC | 11:50 | |
| *** dimtruck is now known as zz_dimtruck | 11:51 | |
| *** esker has joined #openstack-infra | 11:51 | |
| *** markvoelker has quit IRC | 11:51 | |
| *** serg_melikyan has quit IRC | 11:51 | |
| *** zz_dimtruck is now known as dimtruck | 11:51 | |
| *** sergmelikyan has joined #openstack-infra | 11:51 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** amotoki has quit IRC | 11:53 | |
| *** moravec has joined #openstack-infra | 11:53 | |
| *** fawadk has quit IRC | 11:55 | |
| *** ftersin has joined #openstack-infra | 11:56 | |
| *** fungi has quit IRC | 11:57 | |
| *** fungi has joined #openstack-infra | 11:59 | |
| *** dims has joined #openstack-infra | 12:00 | |
| *** dims has quit IRC | 12:00 | |
| *** dims_ has quit IRC | 12:00 | |
| *** dims has joined #openstack-infra | 12:01 | |
| *** esker has quit IRC | 12:01 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Set correct default credentials in the test_backend.sh scripts https://review.openstack.org/213732 | 12:01 |
| *** fdegir has quit IRC | 12:02 | |
| *** dprince has joined #openstack-infra | 12:04 | |
| *** dims has quit IRC | 12:05 | |
| *** dprince has quit IRC | 12:07 | |
| *** smoser has quit IRC | 12:07 | |
| *** dprince has joined #openstack-infra | 12:07 | |
| *** smoser has joined #openstack-infra | 12:07 | |
| *** bharath_ has joined #openstack-infra | 12:08 | |
| *** k4n0 has joined #openstack-infra | 12:09 | |
| *** rfolco has joined #openstack-infra | 12:09 | |
| *** gildub has quit IRC | 12:11 | |
| *** markvoelker has joined #openstack-infra | 12:12 | |
| *** bharath_ has quit IRC | 12:13 | |
| *** fhubik_ has quit IRC | 12:15 | |
| *** fhubik_brb has quit IRC | 12:15 | |
| *** zeih has joined #openstack-infra | 12:15 | |
| *** dims has joined #openstack-infra | 12:15 | |
| *** fhubik has joined #openstack-infra | 12:15 | |
| *** aysyd has joined #openstack-infra | 12:16 | |
| *** jeadr has quit IRC | 12:17 | |
| *** fhubik is now known as fhubik_brb | 12:17 | |
| *** jeadr has joined #openstack-infra | 12:18 | |
| *** salv-orlando has quit IRC | 12:18 | |
| *** bswartz has quit IRC | 12:20 | |
| *** annegentle has joined #openstack-infra | 12:24 | |
| *** gildub has joined #openstack-infra | 12:27 | |
| *** lttrl has joined #openstack-infra | 12:27 | |
| *** mquin has quit IRC | 12:29 | |
| *** mquin has joined #openstack-infra | 12:29 | |
| *** yamamoto_ has quit IRC | 12:29 | |
| *** zhenguo has joined #openstack-infra | 12:29 | |
| *** kgiusti has joined #openstack-infra | 12:33 | |
| *** marcusvrn has joined #openstack-infra | 12:38 | |
| *** shardy_ has joined #openstack-infra | 12:39 | |
| *** svasheka has joined #openstack-infra | 12:40 | |
| *** sobersabre is now known as mvk | 12:40 | |
| *** gildub has quit IRC | 12:40 | |
| *** rbradfor has joined #openstack-infra | 12:40 | |
| *** dims_ has joined #openstack-infra | 12:41 | |
| *** shardy has quit IRC | 12:41 | |
| *** dims has quit IRC | 12:41 | |
| *** ZZelle-off has joined #openstack-infra | 12:42 | |
| svasheka | hi guys, could someone point me in direction of what changes should be made to a job, so it could copy artifacts from test run? | 12:43 |
| *** shardy_ has quit IRC | 12:45 | |
| *** yamamoto has joined #openstack-infra | 12:45 | |
| *** yamamoto has quit IRC | 12:45 | |
| *** shardy has joined #openstack-infra | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:46 | |
| *** enikanorov2 has joined #openstack-infra | 12:46 | |
| *** baoli has joined #openstack-infra | 12:48 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** marcusvrn has quit IRC | 12:49 | |
| *** abregman is now known as abregman|brb | 12:50 | |
| *** enikanorov2_ has joined #openstack-infra | 12:51 | |
| *** enikanorov2_ has quit IRC | 12:51 | |
| *** chlong has joined #openstack-infra | 12:52 | |
| *** baoli has quit IRC | 12:52 | |
| *** enikanorov2_ has joined #openstack-infra | 12:53 | |
| *** enikanorov2 has quit IRC | 12:53 | |
| *** moravec has quit IRC | 12:54 | |
| *** sergmelikyan has quit IRC | 12:56 | |
| *** rlandy has joined #openstack-infra | 12:57 | |
| *** abregman|brb is now known as abregman | 13:00 | |
| *** Sree has quit IRC | 13:01 | |
| *** moravec has joined #openstack-infra | 13:01 | |
| *** dims has joined #openstack-infra | 13:01 | |
| *** enikanorov2_ has quit IRC | 13:02 | |
| openstackgerrit | Sinval Vieira Mendes Neto proposed openstack-infra/project-config: Adding python3-jobs to test python-oneviewclient project https://review.openstack.org/214170 | 13:02 |
| *** dims_ has quit IRC | 13:02 | |
| *** bswartz has joined #openstack-infra | 13:02 | |
| *** tlian has joined #openstack-infra | 13:03 | |
| *** bdemers has joined #openstack-infra | 13:04 | |
| *** yamamoto has joined #openstack-infra | 13:04 | |
| *** abregman_ has joined #openstack-infra | 13:05 | |
| *** wenlock has joined #openstack-infra | 13:06 | |
| *** abregman has quit IRC | 13:07 | |
| *** yamamoto has quit IRC | 13:08 | |
| openstackgerrit | will soula proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin https://review.openstack.org/204615 | 13:08 |
| *** lttrl has left #openstack-infra | 13:11 | |
| *** yamamoto has joined #openstack-infra | 13:12 | |
| *** yamamoto has quit IRC | 13:12 | |
| *** dims_ has joined #openstack-infra | 13:15 | |
| *** swat30 has quit IRC | 13:16 | |
| *** dims has quit IRC | 13:17 | |
| *** swat30 has joined #openstack-infra | 13:17 | |
| *** changbl has joined #openstack-infra | 13:19 | |
| *** persia_ is now known as persia | 13:19 | |
| *** smarcet has joined #openstack-infra | 13:20 | |
| smarcet | fungi: hello, how are u doing? i am having some issues with jekins job config on https://review.openstack.org/#/c/198458/ | 13:20 |
| smarcet | fungi: basically its complains that ext curl for php its not installed | 13:21 |
| *** bexelbie_ has joined #openstack-infra | 13:21 | |
| *** xyang1 has joined #openstack-infra | 13:21 | |
| smarcet | fungi: http://logs.openstack.org/58/198458/21/check/gate-openstackid-unittests/7908166/console.html.gz | 13:22 |
| *** alivigni has joined #openstack-infra | 13:22 | |
| *** abregman_ has quit IRC | 13:22 | |
| *** tiswanso has joined #openstack-infra | 13:23 | |
| *** swat30 has quit IRC | 13:23 | |
| smarcet | fungi: problem is on /usr/local/jenkins/slave_scripts/php-laravel-build.sh, composer should be invoked with --ignore-platform-reqs | 13:23 |
| *** mriedem has joined #openstack-infra | 13:23 | |
| *** cburgess has quit IRC | 13:23 | |
| *** gaurangt has joined #openstack-infra | 13:24 | |
| *** julim has joined #openstack-infra | 13:24 | |
| smarcet | fungi: could i propose that change on https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/php-laravel-build.sh? | 13:24 |
| *** Shrews has quit IRC | 13:24 | |
| *** Shrews has joined #openstack-infra | 13:25 | |
| clarkb | yes please do | 13:25 |
| *** swat30 has joined #openstack-infra | 13:25 | |
| smarcet | clarkb: ok | 13:26 |
| *** cburgess has joined #openstack-infra | 13:26 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Try alternate usernames to get logs from overcloud https://review.openstack.org/214139 | 13:28 |
| *** otter768 has joined #openstack-infra | 13:29 | |
| *** sergmelikyan has joined #openstack-infra | 13:30 | |
| *** dims_ has quit IRC | 13:31 | |
| *** dims has joined #openstack-infra | 13:31 | |
| *** ayoung has joined #openstack-infra | 13:31 | |
| *** yamamoto has joined #openstack-infra | 13:31 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/project-config: Update composer options https://review.openstack.org/214186 | 13:32 |
| *** ftersin has quit IRC | 13:33 | |
| *** mrunge has quit IRC | 13:33 | |
| *** otter768 has quit IRC | 13:34 | |
| pc_m | ianw: ping | 13:34 |
| *** amitgandhinz has joined #openstack-infra | 13:35 | |
| *** abregman_ has joined #openstack-infra | 13:36 | |
| *** Shrews_ has joined #openstack-infra | 13:36 | |
| *** ldnunes has quit IRC | 13:39 | |
| *** prad_ has joined #openstack-infra | 13:39 | |
| *** Shrews_ has quit IRC | 13:40 | |
| *** dims has quit IRC | 13:40 | |
| *** glauco_ has joined #openstack-infra | 13:40 | |
| *** k4n0 has quit IRC | 13:41 | |
| *** Shrews has quit IRC | 13:41 | |
| *** eharney has quit IRC | 13:41 | |
| glauco_ | Hello! I'm currently willing to work implementing integration tests for puppet-gerrit. Is there anyone already working on this? | 13:41 |
| *** dimtruck is now known as zz_dimtruck | 13:41 | |
| *** Shrews has joined #openstack-infra | 13:41 | |
| *** changbl has quit IRC | 13:41 | |
| *** ihrachyshka has quit IRC | 13:43 | |
| *** dims has joined #openstack-infra | 13:43 | |
| *** zz_dimtruck is now known as dimtruck | 13:44 | |
| EmilienM | pabelanger: do you have thoughts about https://review.openstack.org/#/c/213502/ and gate-puppet-keystone-puppet-beaker-rspec-dsvm-centos7 failure? | 13:45 |
| *** jeadr has quit IRC | 13:45 | |
| *** ihrachyshka has joined #openstack-infra | 13:46 | |
| *** jeadr has joined #openstack-infra | 13:46 | |
| *** fhubik_brb is now known as fhubik | 13:47 | |
| *** deepakcs has quit IRC | 13:48 | |
| *** ldnunes has joined #openstack-infra | 13:48 | |
| *** fhubik has quit IRC | 13:50 | |
| *** tvanderwiel has joined #openstack-infra | 13:50 | |
| *** claudiub has quit IRC | 13:51 | |
| *** kushal has joined #openstack-infra | 13:51 | |
| *** matbu has joined #openstack-infra | 13:52 | |
| *** marcusvrn_ has joined #openstack-infra | 13:52 | |
| *** tvanderwiel has left #openstack-infra | 13:53 | |
| *** bharath has joined #openstack-infra | 13:54 | |
| openstackgerrit | Kurt Taylor proposed openstack-infra/infra-specs: Host an initial CI monitoring dashboard service https://review.openstack.org/194437 | 13:54 |
| *** claudiub has joined #openstack-infra | 13:55 | |
| *** dizquierdo has joined #openstack-infra | 13:55 | |
| *** sergmelikyan has quit IRC | 13:56 | |
| *** dkranz has joined #openstack-infra | 13:58 | |
| *** armax has joined #openstack-infra | 14:00 | |
| *** tiswanso has quit IRC | 14:00 | |
| *** sergmelikyan has joined #openstack-infra | 14:00 | |
| glauco_ | @crinkle are you working on puppet-gerrit? | 14:01 |
| *** tiswanso has joined #openstack-infra | 14:01 | |
| *** tchaypo has joined #openstack-infra | 14:02 | |
| *** jvrbanac has quit IRC | 14:02 | |
| *** ramishra has joined #openstack-infra | 14:02 | |
| *** dimtruck is now known as zz_dimtruck | 14:02 | |
| *** zeih has quit IRC | 14:02 | |
| *** DuncanT has joined #openstack-infra | 14:03 | |
| *** dims has quit IRC | 14:04 | |
| *** otherwiseguy has quit IRC | 14:04 | |
| *** dims has joined #openstack-infra | 14:05 | |
| *** enikanorov2 has joined #openstack-infra | 14:05 | |
| *** barnaby has quit IRC | 14:06 | |
| *** nadya has quit IRC | 14:07 | |
| *** fdegir has joined #openstack-infra | 14:07 | |
| *** jvrbanac has joined #openstack-infra | 14:09 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:11 | |
| *** esker has joined #openstack-infra | 14:12 | |
| HenryG | lifeless: ping | 14:12 |
| fungi | clarkb: jeblair: mordred: pleia2: jhesketh: SergeyLukjanov: here's some fun for a tuesday... someone coming from thousands of random ip addresses within china mobile is spamming all our mailing lists with spoofed subscription requests for randomized gmail addresses | 14:13 |
| jhesketh | fungi: oh joy | 14:13 |
| fungi | this is going to be fun to figure out how to prevent | 14:13 |
| SergeyLukjanov | fungi, morning | 14:14 |
| jhesketh | fungi: any particular lists? | 14:14 |
| fungi | jhesketh: yes, all of them | 14:14 |
| fungi | as far as i can tell | 14:14 |
| *** eharney has joined #openstack-infra | 14:14 | |
| fungi | sudo grep 'kemo\.mart' /var/log/mailman/subscribe|less | 14:15 |
| fungi | if you want to see specifics | 14:15 |
| jhesketh | fungi: I didn't realise the lists were in infra.. or are they? | 14:15 |
| fungi | jhesketh: yep, lists.openstack.org is the server. runs mailman on ubuntu precise | 14:15 |
| *** abregman_ is now known as abregman | 14:15 | |
| jhesketh | right, I guess I've just never seen/touched any config there | 14:15 |
| *** ashleighfarnham has joined #openstack-infra | 14:15 | |
| SergeyLukjanov | fungi, we have a single server for mailing lists with a dns record using the rackspace dns service, am I correct? | 14:15 |
| *** zeih has joined #openstack-infra | 14:16 | |
| * jhesketh is blind, there it is | 14:16 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Separate tests in separate files and classes https://review.openstack.org/213521 | 14:16 |
| fungi | SergeyLukjanov: right | 14:16 |
| *** otherwiseguy has joined #openstack-infra | 14:16 | |
| SergeyLukjanov | probably temp DENY for the whole subnet for the attacked URI? | 14:17 |
| *** ZZelle-off has quit IRC | 14:18 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: WIP Add support for listing folder indexes https://review.openstack.org/214207 | 14:19 |
| jhesketh | clarkb: ^ if you get a chance to read that sometime that'd be awesome. I still need to write real tests for it, but I'm heading to bed | 14:19 |
| fungi | SergeyLukjanov: well, it's not just one network... it looks like it's scattered all over china mobile's ip space (>5000 ip addresses) | 14:20 |
| *** peristeri has quit IRC | 14:20 | |
| *** ashleighfarnham has quit IRC | 14:20 | |
| jhesketh | fungi, SergeyLukjanov: if they are both random IP's and random addresses I'm not sure what else we could do | 14:20 |
| *** annegentle has quit IRC | 14:20 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Adds Cloudbees folder plugin support https://review.openstack.org/180185 | 14:20 |
| *** fhubik has joined #openstack-infra | 14:21 | |
| fungi | i count 181 out of 256 possible values in the first octet alone | 14:22 |
| *** gaurangt has left #openstack-infra | 14:22 | |
| jhesketh | :-( | 14:22 |
| fungi | so it's scattered over 70% of the ipv4 namespace by first octet | 14:22 |
| *** barnaby has joined #openstack-infra | 14:23 | |
| SergeyLukjanov | fungi, heh... | 14:23 |
| fungi | china mobile apparently has lots of tiny ip networks everywhere | 14:23 |
| jhesketh | fungi: are you sure they are all china mobile? | 14:23 |
| clarkb | how convenient | 14:23 |
| *** FallenPegasus has joined #openstack-infra | 14:23 | |
| fungi | jhesketh: the ones i spot-checked all were, but i didn't whois all 5132 of them yet | 14:24 |
| jhesketh | heh | 14:24 |
| jhesketh | fungi: I assume you checked a few with different first octets | 14:24 |
| fungi | it's entirely possible that this is coming from a botnet and only the majority of the addresses are within china mobile | 14:25 |
| *** fhubik_ has joined #openstack-infra | 14:25 | |
| *** Shrews has quit IRC | 14:25 | |
| jhesketh | right, that's what I was suspecting | 14:25 |
| fungi | jhesketh: yeah, for example 39.171.66.82 vs 117.166.26.241 | 14:25 |
| jhesketh | an odd thing to target though | 14:25 |
| fungi | jhesketh: i'm wondering if it might be a reflection attack | 14:25 |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Migration to using requests https://review.openstack.org/205637 | 14:26 |
| *** Shrews has joined #openstack-infra | 14:26 | |
| *** [HeOS] has joined #openstack-infra | 14:26 | |
| fungi | jhesketh: the subscription confirmation e-mail is much larger than the subscription request, so there's certainly opportunity for amplification | 14:26 |
| *** HeOS has quit IRC | 14:26 | |
| *** prad_ is now known as pradk | 14:26 | |
| fungi | somewhat larger anyway, i don't know how much, but regardless if you have a large bot army you don't necessarily need amplification | 14:26 |
| clarkb | are we seeing it affect the service? | 14:27 |
| *** jeadr has quit IRC | 14:27 | |
| *** afazkas has quit IRC | 14:27 | |
| fungi | reflection attacks are often either because you want to hide the origin of your attacks or because you want to cast blame on another party | 14:27 |
| *** jeadr has joined #openstack-infra | 14:28 | |
| fungi | clarkb: the only impact reported so far is vetted-subscription list moderators having to delete hundreds of pending requests | 14:28 |
| fungi | though it's possible gmail will start blocking messages from lists.openstack.org eventually, no idea | 14:29 |
| *** zul has joined #openstack-infra | 14:29 | |
| *** salv-orlando has joined #openstack-infra | 14:30 | |
| *** wshao has joined #openstack-infra | 14:30 | |
| eantyshev | SergeyLukjanov: Hello! can I ask you to review this change for system-config: https://review.openstack.org/212554 | 14:30 |
| fungi | we currently have 5665 deferred deliveries for addresses matching the patter of the attack sitting in the exim outbound queue on lists.o.o | 14:30 |
| *** fhubik is now known as fhubik_brb | 14:30 | |
| fungi | s/patter/pattern/ | 14:30 |
| *** [HeOS] has quit IRC | 14:31 | |
| fungi | i expect they're all subscription auto responses | 14:31 |
| *** tonytan4ever has joined #openstack-infra | 14:31 | |
| *** zz_dimtruck is now known as dimtruck | 14:31 | |
| *** kiran-r has joined #openstack-infra | 14:31 | |
| *** kiran-r has quit IRC | 14:31 | |
| *** vlaza has left #openstack-infra | 14:31 | |
| *** sdake has joined #openstack-infra | 14:31 | |
| *** kiran-r has joined #openstack-infra | 14:31 | |
| *** stevemar has joined #openstack-infra | 14:32 | |
| *** kiran-r has quit IRC | 14:33 | |
| fungi | gmail is temporarily rejecting them with a message like "The user you are trying to contact is receiving mail at a rate that prevents additional messages from being delivered." | 14:33 |
| *** kiran-r has joined #openstack-infra | 14:33 | |
| clarkb | can we temporarily kill new subscriptions? | 14:33 |
| *** salv-orlando has quit IRC | 14:34 | |
| *** sabeen1 has joined #openstack-infra | 14:35 | |
| *** dims_ has joined #openstack-infra | 14:35 | |
| fungi | i'm not sure. reading up on options | 14:36 |
| *** romcheg has joined #openstack-infra | 14:36 | |
| *** spzala has joined #openstack-infra | 14:37 | |
| romcheg | Hi folks! | 14:37 |
| *** mrunge has joined #openstack-infra | 14:37 | |
| *** dims has quit IRC | 14:37 | |
| romcheg | What version of tox is installed on CI nodes? | 14:37 |
| *** ftersin has joined #openstack-infra | 14:37 | |
| *** zeih has quit IRC | 14:37 | |
| *** sdake_ has joined #openstack-infra | 14:37 | |
| *** sabeen3 has joined #openstack-infra | 14:37 | |
| *** stevemar has quit IRC | 14:39 | |
| *** sabeen1 has quit IRC | 14:39 | |
| *** bexelbie_ is now known as bexelbie | 14:39 | |
| *** stevemar has joined #openstack-infra | 14:39 | |
| *** thcipriani|afk is now known as thcipriani | 14:40 | |
| *** sdake has quit IRC | 14:40 | |
| *** spzala has quit IRC | 14:42 | |
| *** glauco_ has quit IRC | 14:42 | |
| crinkle | glauco_: I haven't been working on tests for puppet-gerrit, happy to help out though! | 14:43 |
| *** mquin_ has joined #openstack-infra | 14:44 | |
| *** annegentle has joined #openstack-infra | 14:44 | |
| *** mquin has quit IRC | 14:47 | |
| *** enikanorov2 has quit IRC | 14:48 | |
| *** tjones has joined #openstack-infra | 14:48 | |
| *** glauco_ has joined #openstack-infra | 14:50 | |
| *** regXboi has joined #openstack-infra | 14:50 | |
| *** jeadr has quit IRC | 14:50 | |
| *** julim_ has joined #openstack-infra | 14:51 | |
| *** bdemers has quit IRC | 14:51 | |
| *** dangers_away is now known as dangers | 14:51 | |
| *** jeadr has joined #openstack-infra | 14:51 | |
| *** madong_ has joined #openstack-infra | 14:52 | |
| *** angdraug has joined #openstack-infra | 14:53 | |
| *** jkt_ has joined #openstack-infra | 14:54 | |
| *** erlon has joined #openstack-infra | 14:54 | |
| *** zul has quit IRC | 14:54 | |
| *** fdegir_ has joined #openstack-infra | 14:55 | |
| *** mtanino has joined #openstack-infra | 14:55 | |
| *** wshao has quit IRC | 14:55 | |
| *** Kiall_ has joined #openstack-infra | 14:57 | |
| *** bswartz has quit IRC | 14:57 | |
| *** michchap_ has joined #openstack-infra | 14:58 | |
| *** dan_ has joined #openstack-infra | 14:58 | |
| *** bdemers has joined #openstack-infra | 14:58 | |
| *** dan_ is now known as Guest34051 | 14:58 | |
| *** mquin_ has quit IRC | 14:59 | |
| *** stevemar has quit IRC | 14:59 | |
| *** tonytan4ever has quit IRC | 14:59 | |
| *** barnaby has quit IRC | 14:59 | |
| *** fdegir has quit IRC | 14:59 | |
| *** julim has quit IRC | 14:59 | |
| *** dprince has quit IRC | 14:59 | |
| *** larainema has quit IRC | 14:59 | |
| *** Kiall has quit IRC | 14:59 | |
| *** rcarrillocruz has quit IRC | 14:59 | |
| *** thorst has quit IRC | 14:59 | |
| *** jtomasek has quit IRC | 14:59 | |
| *** pcrews has quit IRC | 14:59 | |
| *** ilyashakhat has quit IRC | 14:59 | |
| *** skraynev has quit IRC | 14:59 | |
| *** michchap has quit IRC | 14:59 | |
| *** jkt has quit IRC | 14:59 | |
| *** leifmadsen_ has quit IRC | 14:59 | |
| *** cody-somerville- has quit IRC | 14:59 | |
| *** milligana1 has quit IRC | 14:59 | |
| *** wznoinsk has quit IRC | 14:59 | |
| *** dan has quit IRC | 14:59 | |
| *** fdegir_ is now known as fdegir | 14:59 | |
| *** Guest34051 is now known as dan | 14:59 | |
| *** dims has joined #openstack-infra | 15:01 | |
| *** leifmadsen has joined #openstack-infra | 15:01 | |
| *** mquin has joined #openstack-infra | 15:01 | |
| *** skolekonov has joined #openstack-infra | 15:02 | |
| *** yamahata has joined #openstack-infra | 15:02 | |
| *** austin81 has joined #openstack-infra | 15:02 | |
| *** achanda has joined #openstack-infra | 15:03 | |
| *** hdd has joined #openstack-infra | 15:03 | |
| *** dims_ has quit IRC | 15:03 | |
| *** rcarrillocruz has joined #openstack-infra | 15:03 | |
| *** julim has joined #openstack-infra | 15:04 | |
| *** smccully has joined #openstack-infra | 15:04 | |
| *** achanda has quit IRC | 15:05 | |
| *** dolphm has joined #openstack-infra | 15:05 | |
| *** stevemar has joined #openstack-infra | 15:05 | |
| *** enikanorov2 has joined #openstack-infra | 15:06 | |
| *** julim_ has quit IRC | 15:06 | |
| *** claudiub has quit IRC | 15:07 | |
| *** amotoki has joined #openstack-infra | 15:08 | |
| *** doug-fish has joined #openstack-infra | 15:08 | |
| *** fhubik_ has quit IRC | 15:09 | |
| *** fhubik_brb has quit IRC | 15:09 | |
| *** flepied has quit IRC | 15:09 | |
| *** zul has joined #openstack-infra | 15:09 | |
| *** ashleighfarnham has joined #openstack-infra | 15:10 | |
| fungi | clarkb: jeblair: mordred: pleia2: jhesketh: SergeyLukjanov: on a review of the logs, it appears that the incident ran from 09:27 utc yesterday to 08:45 utc today but no new hits for 5+ hours now. i've deleted all the deferred messages for the involved addresses from the exim outbound queue | 15:11 |
| jhesketh | well that's good I guess | 15:11 |
| *** tonytan4ever has joined #openstack-infra | 15:12 | |
| *** dprince has joined #openstack-infra | 15:12 | |
| jhesketh | nice work anyway | 15:12 |
| *** thorst has joined #openstack-infra | 15:12 | |
| *** jtomasek has joined #openstack-infra | 15:12 | |
| *** ilyashakhat has joined #openstack-infra | 15:12 | |
| *** skraynev has joined #openstack-infra | 15:12 | |
| *** cody-somerville- has joined #openstack-infra | 15:12 | |
| *** wznoinsk has joined #openstack-infra | 15:12 | |
| jhesketh | I should head to bed, there's a meeting soon | 15:12 |
| jhesketh | sorry I couldn't help more! | 15:12 |
| clarkb | jhesketh: I will look between meetings today | 15:12 |
| fungi | night! | 15:12 |
| jhesketh | clarkb: all good, not urgent (also not entirely finished) | 15:12 |
| jhesketh | night all | 15:12 |
| *** milligana has joined #openstack-infra | 15:13 | |
| clarkb | good night | 15:13 |
| *** ashleighfarnham has quit IRC | 15:14 | |
| *** barnaby has joined #openstack-infra | 15:15 | |
| *** sabeen3 has quit IRC | 15:16 | |
| *** dprince has quit IRC | 15:16 | |
| *** sabeen has joined #openstack-infra | 15:16 | |
| *** bswartz has joined #openstack-infra | 15:17 | |
| *** arxcruz has quit IRC | 15:18 | |
| *** woodster_ has joined #openstack-infra | 15:18 | |
| *** dannywilson has joined #openstack-infra | 15:18 | |
| clarkb | I am going to remove the bandersnatch state files on the other three pypi mirrors to get them resyncing today | 15:19 |
| clarkb | doing that now | 15:19 |
| clarkb | (dfw looks happy) | 15:19 |
| *** tsekiyama has joined #openstack-infra | 15:19 | |
| clarkb | now to remember the hpcloud hostname | 15:21 |
| notnownikki | clarkb, ah, that got you too? I've been doing that today :) | 15:22 |
| *** ajmiller_ has quit IRC | 15:22 | |
| *** enikanorov2 has quit IRC | 15:23 | |
| *** ociuhandu has joined #openstack-infra | 15:23 | |
| *** dannywilson has quit IRC | 15:23 | |
| *** dannywil_ has joined #openstack-infra | 15:23 | |
| *** glauco_ has quit IRC | 15:23 | |
| clarkb | notnownikki: yes it was a bug in pypi that broke the mirror protocol | 15:24 |
| clarkb | notnownikki: according to dstufft a resync is the only way to correct now that pypi is fixed | 15:24 |
| *** jeadr has quit IRC | 15:24 | |
| *** dannywil_ has quit IRC | 15:24 | |
| clarkb | pypi.region-1.geo-1.openstack.org ftr | 15:25 |
| *** jeadr has joined #openstack-infra | 15:25 | |
| *** dannywilson has joined #openstack-infra | 15:25 | |
| notnownikki | clarkb, yeah, I look after the mirroring for 4 of our labs and the one in the public cloud, lots of fun getting them going again without waiting for the time to do a resync that would take an age on some of the lower bandwidth sites | 15:25 |
| zul | EmilienM: ping | 15:26 |
| *** puranamr has joined #openstack-infra | 15:26 | |
| EmilienM | zul: pong | 15:26 |
| EmilienM | zul: presently in puppet meeting | 15:26 |
| zul | EmilienM: whats broken in heat? | 15:26 |
| EmilienM | but about to end | 15:26 |
| EmilienM | zul: let me show you | 15:27 |
| *** zz_ja has quit IRC | 15:28 | |
| EmilienM | zul: http://logs.openstack.org/90/209290/6/check/gate-puppet-heat-puppet-beaker-rspec-dsvm-trusty/0b8542c/logs/heat/heat-engine.txt.gz | 15:28 |
| *** dims has quit IRC | 15:29 | |
| EmilienM | zul: http://logs.openstack.org/90/209290/6/check/gate-puppet-heat-puppet-beaker-rspec-dsvm-trusty/0b8542c/console.html#_2015-08-18_13_58_07_018 | 15:29 |
| zul | EmilienM: ah ok | 15:29 |
| *** maurosr has quit IRC | 15:29 | |
| *** dims has joined #openstack-infra | 15:30 | |
| *** otter768 has joined #openstack-infra | 15:30 | |
| EmilienM | zul: I already reported that to jamespage, heat/db/sqlalchemy/migrate_repo/migrate.cfg is missing in the package | 15:30 |
| zul | EmilienM: ok cool...ill take a look | 15:30 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Add 'push' and 'tag' github webhook events. https://review.openstack.org/191207 | 15:31 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Support GitHub PR webhooks https://review.openstack.org/163117 | 15:31 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Fix job hierarchy bug. https://review.openstack.org/192457 | 15:31 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Add basic Github Zuul Reporter. https://review.openstack.org/191312 | 15:31 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Allow list values in template parameters. https://review.openstack.org/191208 | 15:31 |
| *** barnaby is now known as bcourt | 15:31 | |
| *** _kiran_ has joined #openstack-infra | 15:32 | |
| *** kiran-r has quit IRC | 15:33 | |
| *** salv-orlando has joined #openstack-infra | 15:33 | |
| *** bpokorny has joined #openstack-infra | 15:33 | |
| mordred | fungi: wow fun! | 15:35 |
| *** otter768 has quit IRC | 15:35 | |
| *** zz_ja has joined #openstack-infra | 15:36 | |
| *** woodster_ has quit IRC | 15:36 | |
| *** ramishra has quit IRC | 15:36 | |
| *** DuncanT has quit IRC | 15:36 | |
| *** tchaypo has quit IRC | 15:36 | |
| *** marcusvrn_ has quit IRC | 15:36 | |
| *** erlon has quit IRC | 15:36 | |
| *** maurosr has joined #openstack-infra | 15:36 | |
| *** btully has joined #openstack-infra | 15:37 | |
| *** dkranz has quit IRC | 15:37 | |
| mordred | rcarrillocruz: around? | 15:40 |
| *** Swami has joined #openstack-infra | 15:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface https://review.openstack.org/214243 | 15:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move keystone to common identity client interface https://review.openstack.org/214244 | 15:40 |
| *** jeadr has quit IRC | 15:40 | |
| clarkb | mordred asselin was having shade/oscc trouble yesterday | 15:40 |
| clarkb | may be worth catching up on that | 15:40 |
| *** jeadr has joined #openstack-infra | 15:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move keystone to common identity client interface https://review.openstack.org/214244 | 15:41 |
| mordred | cool | 15:41 |
| clarkb | was getting keystone errors saying user foo is does not have permissions to tenant bar | 15:42 |
| *** rm_work is now known as rm_work|away | 15:43 | |
| mordred | yeah- was just reading the scrollback | 15:44 |
| *** dims has quit IRC | 15:45 | |
| *** zul has quit IRC | 15:46 | |
| *** spzala has joined #openstack-infra | 15:46 | |
| *** geoffarnold has joined #openstack-infra | 15:46 | |
| mordred | asselin: so - it does not work for your admin user and does work for your non-admin user? | 15:46 |
| clarkb | mordred yup and is juno cloud | 15:47 |
| *** dims has joined #openstack-infra | 15:47 | |
| mordred | hrm | 15:47 |
| mordred | Shrews: ^^ | 15:47 |
| mordred | Shrews: we has strange bug | 15:47 |
| *** btully has quit IRC | 15:48 | |
| *** erlon has joined #openstack-infra | 15:49 | |
| *** _kiran_ has quit IRC | 15:50 | |
| *** geoffarnold has quit IRC | 15:50 | |
| *** bexelbie has quit IRC | 15:50 | |
| *** geoffarnold has joined #openstack-infra | 15:50 | |
| *** ashleighfarnham has joined #openstack-infra | 15:51 | |
| *** sergmelikyan has quit IRC | 15:51 | |
| *** pabelanger has quit IRC | 15:51 | |
| *** afazkas has joined #openstack-infra | 15:51 | |
| openstackgerrit | Merged openstack-infra/system-config: Support CloudLinux in install_puppet.sh https://review.openstack.org/212554 | 15:52 |
| zaro | morning | 15:53 |
| clarkb | ok bandersnatch is syncing now | 15:53 |
| *** mestery has joined #openstack-infra | 15:54 | |
| *** abregman has quit IRC | 15:55 | |
| *** nadya has joined #openstack-infra | 15:55 | |
| *** sthillma has joined #openstack-infra | 15:56 | |
| mordred | woot | 15:56 |
| *** ashleighfarnham has quit IRC | 15:56 | |
| *** sergmelikyan has joined #openstack-infra | 15:56 | |
| *** dkranz has joined #openstack-infra | 15:57 | |
| *** bdemers has quit IRC | 15:57 | |
| *** sthillma has quit IRC | 15:57 | |
| *** eharney has quit IRC | 15:58 | |
| *** marcusvrn_ has joined #openstack-infra | 15:58 | |
| *** ajmiller has joined #openstack-infra | 15:58 | |
| *** eharney has joined #openstack-infra | 15:59 | |
| clarkb | thing I notice about elasticsearch slowness, we have multimegabyte unittest log files | 16:00 |
| clarkb | I am willing to bet that we feed some very large log files into logstash and the whole thing slows down | 16:00 |
| *** ZZelle has joined #openstack-infra | 16:02 | |
| *** angdraug has quit IRC | 16:02 | |
| *** ZZelle is now known as ZZelle-off | 16:02 | |
| *** glauco_ has joined #openstack-infra | 16:03 | |
| *** woodster_ has joined #openstack-infra | 16:03 | |
| *** bswartz has quit IRC | 16:04 | |
| *** tjones has quit IRC | 16:04 | |
| *** bswartz has joined #openstack-infra | 16:06 | |
| *** vahidh has joined #openstack-infra | 16:08 | |
| *** ihrachyshka has quit IRC | 16:08 | |
| clarkb | looks like we lost a few of the elasticsearch client processes on the worker nodes | 16:08 |
| clarkb | and the logs were indeed stuck at neutron unittests | 16:08 |
| *** jistr has quit IRC | 16:08 | |
| *** BharatK has quit IRC | 16:10 | |
| *** tchaypo has joined #openstack-infra | 16:11 | |
| *** ramishra has joined #openstack-infra | 16:12 | |
| *** DuncanT has joined #openstack-infra | 16:12 | |
| *** jkt_ is now known as jkt | 16:13 | |
| fungi | clarkb: also there's been some scheduled network maintenance in rackspace in the past few days, one of which seemed to cause one of my personal instances to lose contact with a cinder volume and mark that filesystem read-only | 16:13 |
| *** romcheg has quit IRC | 16:13 | |
| fungi | so keep an eye out for similar problems on our systems | 16:13 |
| clarkb | fungi: will dothanks | 16:13 |
| clarkb | in this case we seem to have just dos'd ourselves with larger logfiles | 16:13 |
| *** tjones has joined #openstack-infra | 16:14 | |
| fungi | certainly sounds that way | 16:14 |
| clarkb | I am going through and restarting es daemons as necessary and cleaning up any gearman workers | 16:14 |
| clarkb | as an improvement to the workers we can work on having them do partial http gets and process smaller portions of the files at a time | 16:14 |
| clarkb | also I imagine this was buffer bbackup bloating in repsonse to the cluster going down | 16:17 |
| *** ashleighfarnham has joined #openstack-infra | 16:17 | |
| *** flepied has joined #openstack-infra | 16:17 | |
| *** jcoufal has quit IRC | 16:20 | |
| *** akraminsky has joined #openstack-infra | 16:20 | |
| zaro | pelix: ping | 16:21 |
| *** esmute_ has joined #openstack-infra | 16:22 | |
| pelix | zaro: pong | 16:22 |
| *** alexsyip has joined #openstack-infra | 16:23 | |
| zaro | pelix: reviewing folders support. | 16:23 |
| zaro | pelix: does it seem odd to you that you need to use the jobs APIs to create a folder? | 16:24 |
| zaro | pelix: shouldn't there be a create_folder()/delete_folder()/update_folder() set of APIs? something similar to views? | 16:24 |
| pelix | yes, but since it's more of an oddity within Jenkins I'm not sure if we can easily treat it the same as views. I guess we could look to wrap the calls to the jobs api with some additional checks that they are folders, etc. but as far as Jenkins is concerned, folders are just another type of job | 16:26 |
| *** unicell has quit IRC | 16:26 | |
| *** geoffarnold has quit IRC | 16:27 | |
| *** rm_work|away is now known as rm_work | 16:27 | |
| *** romcheg has joined #openstack-infra | 16:27 | |
| zaro | pelix: ok, i've never used folders. from a conceptual context i associated it with views but looks like it's implemented like jobs. | 16:28 |
| *** tjones has quit IRC | 16:30 | |
| zaro | so i think having an API to decouple folder from jobs would be nice but not necessary for this change. | 16:30 |
| *** tjones has joined #openstack-infra | 16:30 | |
| *** sthillma has joined #openstack-infra | 16:30 | |
| *** sthillma_ has joined #openstack-infra | 16:31 | |
| *** doug-fish has quit IRC | 16:32 | |
| *** doug-fish has joined #openstack-infra | 16:33 | |
| *** devvesa has quit IRC | 16:34 | |
| *** sdake_ is now known as sdake | 16:34 | |
| *** esker has quit IRC | 16:34 | |
| *** pcrews has joined #openstack-infra | 16:35 | |
| *** sthillma has quit IRC | 16:35 | |
| *** sthillma_ is now known as sthillma | 16:35 | |
| *** wshao has joined #openstack-infra | 16:36 | |
| *** dizquierdo has quit IRC | 16:36 | |
| *** sputnik13 has joined #openstack-infra | 16:36 | |
| *** romcheg1 has joined #openstack-infra | 16:36 | |
| *** dims has quit IRC | 16:36 | |
| *** romcheg has quit IRC | 16:37 | |
| *** scheuran has quit IRC | 16:38 | |
| clarkb | ok I think I have cleaned up the indexing situation, we will have to see how it does over the next day or so | 16:39 |
| *** dims has joined #openstack-infra | 16:39 | |
| *** dprince has joined #openstack-infra | 16:40 | |
| *** annegentle has quit IRC | 16:41 | |
| *** afazkas has quit IRC | 16:42 | |
| *** krtaylor has quit IRC | 16:43 | |
| *** regXboi has quit IRC | 16:45 | |
| *** doug-fish has quit IRC | 16:46 | |
| *** doug-fish has joined #openstack-infra | 16:46 | |
| *** sputnik13 has quit IRC | 16:47 | |
| *** gothicmi1dfood has joined #openstack-infra | 16:47 | |
| *** gothicmi1dfood has quit IRC | 16:48 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: OpenstackId resource server https://review.openstack.org/178853 | 16:48 |
| *** gothicmindfood has quit IRC | 16:48 | |
| *** gothicmindfood has joined #openstack-infra | 16:48 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 16:49 |
| *** wshao has quit IRC | 16:49 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:50 | |
| *** dizquierdo has joined #openstack-infra | 16:51 | |
| *** davideagnello has joined #openstack-infra | 16:51 | |
| *** sshnaidm has quit IRC | 16:53 | |
| *** sputnik13 has joined #openstack-infra | 16:54 | |
| *** dims has quit IRC | 16:54 | |
| *** unicell has joined #openstack-infra | 16:55 | |
| *** shardy_ has joined #openstack-infra | 16:55 | |
| *** shardy has quit IRC | 16:55 | |
| *** krtaylor has joined #openstack-infra | 16:56 | |
| sthillma | Hi all, any chance we can get someone else to look at https://review.openstack.org/#/c/212221 and get it approved (just adding create premissions for n1k-vsm stackforge)? thanks in advance | 16:57 |
| *** dims has joined #openstack-infra | 16:57 | |
| asselin_ | mordred, Shrews clarkb yes wasn't working, even with curl the command directly to the juno openstack backend. | 16:57 |
| greghaynes | SpamapS: ok, I have successfully gotten the scoop on how to make bifrost do the vlan thing we need from TheJulia | 16:57 |
| greghaynes | mission accomplished | 16:57 |
| asselin_ | mordred, Shrews clarkb so I think the json or endpoint or something changed that's incompatible. I can login via horizon just fine | 16:58 |
| *** shashank_hegde has joined #openstack-infra | 16:58 | |
| *** ifarkas has quit IRC | 16:58 | |
| SpamapS | greghaynes: fantastic | 16:58 |
| *** kushal has quit IRC | 16:59 | |
| SpamapS | greghaynes: I've written some code that transforms a fully populated gh2bf inventory into an erb template with hostnames assigned... | 16:59 |
| *** sputnik1_ has joined #openstack-infra | 16:59 | |
| sdague | clarkb: any chance I can ask a favor? devstack is basically broken for everyone because it's not running in constraints mode | 16:59 |
| greghaynes | best file format name ever | 16:59 |
| sdague | I did one more rev of your patch, but connectivity is touch / go at linux con | 16:59 |
| clarkb | sdague: so need me to pick that up again? | 17:00 |
| *** sergmelikyan has quit IRC | 17:00 | |
| sdague | and I don't have a great local test env | 17:00 |
| *** sputnik13 has quit IRC | 17:00 | |
| sdague | yeh, if it fails tests this time | 17:00 |
| clarkb | ok will keep an eye on it | 17:00 |
| sdague | I think it's mostly just variables not in the right scope | 17:00 |
| *** sabeen has quit IRC | 17:00 | |
| *** shardy_ has quit IRC | 17:00 | |
| greghaynes | clarkb: oh hey, were all starting to pounce on an etherpad FYI | 17:00 |
| greghaynes | clarkb: we'll see if it holds up this time... | 17:00 |
| sdague | and right now we just need a thing that works, we can make it pretty in the future | 17:01 |
| *** dizquierdo has quit IRC | 17:01 | |
| *** shardy has joined #openstack-infra | 17:01 | |
| clarkb | greghaynes: exciting | 17:01 |
| clarkb | it is running with the fixes since vancouver | 17:01 |
| greghaynes | fun | 17:01 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add tools to make infra-cloud inventory templates https://review.openstack.org/214279 | 17:01 |
| SpamapS | greghaynes: ^^ scripts for that | 17:02 |
| *** mtrovo has joined #openstack-infra | 17:02 | |
| clarkb | hogepodge: did switching away from safari fix your eitherpad issues? | 17:02 |
| clarkb | greghaynes: if people do have trouble om macs ask them to use a different brwoser | 17:02 |
| SpamapS | greghaynes: I ran inventory-full through that and came out with an erb template + hiera.yaml that I think might actually work. ;) | 17:02 |
| *** ajmiller has quit IRC | 17:02 | |
| *** dims_ has joined #openstack-infra | 17:02 | |
| greghaynes | SpamapS: whoa | 17:03 |
| greghaynes | famous last words | 17:03 |
| *** dims has quit IRC | 17:04 | |
| SpamapS | greghaynes: I figure we can keep those as manual-ish tools since we'll only use them when we need to process new inventory. | 17:04 |
| *** nadya has quit IRC | 17:05 | |
| *** luqas has joined #openstack-infra | 17:05 | |
| *** zhenguo has quit IRC | 17:05 | |
| *** jerryz has joined #openstack-infra | 17:06 | |
| *** hyakuhei has joined #openstack-infra | 17:06 | |
| *** mestery has quit IRC | 17:07 | |
| *** markus_z has quit IRC | 17:07 | |
| *** madong_ has quit IRC | 17:07 | |
| *** sputnik1_ has quit IRC | 17:07 | |
| *** madong_ has joined #openstack-infra | 17:08 | |
| *** sputnik13 has joined #openstack-infra | 17:08 | |
| *** akraminsky has quit IRC | 17:08 | |
| fungi | fwiw the etherpad server is a tad loaded. average is 2-3 | 17:08 |
| SpamapS | ops meetup? | 17:08 |
| fungi | looks like there's a runaway abiword process eating a whole cpu for quite some time | 17:08 |
| clarkb | fungi: yes ops meetup | 17:08 |
| clarkb | fungi: you can safelykill it | 17:08 |
| fungi | right, ops meetup is why i was looking at the server | 17:08 |
| *** dims has joined #openstack-infra | 17:08 | |
| *** dims_ has quit IRC | 17:09 | |
| clarkb | fungi: I see that we might consider a bigger host before the summit | 17:09 |
| clarkb | ENOMEMORY | 17:09 |
| greghaynes | Its likely the ops summit is larger than summit load | 17:09 |
| greghaynes | since its everyone on one pad | 17:09 |
| fungi | yeah, was just starting to poke around cacti stats for it | 17:09 |
| clarkb | greghaynes: in the past ops at summit has been worst | 17:10 |
| clarkb | greghaynes: so possible | 17:10 |
| SpamapS | one pad == one process or something? | 17:10 |
| *** akraminsky has joined #openstack-infra | 17:10 | |
| clarkb | SpamapS: its all one process | 17:10 |
| clarkb | SpamapS: because node | 17:10 |
| *** sputnik13 has quit IRC | 17:10 | |
| fungi | possible they also all originate from a single ip address | 17:10 |
| SpamapS | oh right, node.. :-P | 17:10 |
| clarkb | greghaynes: looking at cacti this isn't really all that much worse than before though | 17:10 |
| *** pcrews has quit IRC | 17:10 | |
| greghaynes | SpamapS: n^2 problem :p | 17:10 |
| *** skolekonov has quit IRC | 17:10 | |
| *** kiran-r has joined #openstack-infra | 17:11 | |
| greghaynes | everyone has to see updates who are on the pad | 17:11 |
| fungi | oh yeah, the used memory is riding very close to system max. not much for cache/buffers | 17:11 |
| *** dims_ has joined #openstack-infra | 17:11 | |
| * SpamapS adds another jenga block on the top | 17:11 | |
| fungi | swap usage has crept up to ~300mib | 17:11 |
| SpamapS | IMO, swap usage on production servers == eminent disaster. | 17:12 |
| *** minwang2 has joined #openstack-infra | 17:12 | |
| fungi | meh, some swap is expected | 17:12 |
| SpamapS | Except that usual "this process never does anything" swap | 17:12 |
| greghaynes | SpamapS: The hiera.yaml you speak of, what all goes in there? | 17:12 |
| clarkb | SpamapS: I think that is what is happening here | 17:12 |
| clarkb | node is happily running in memory and abiword is off in swap derping | 17:12 |
| SpamapS | greghaynes: hostname_ipmi_password | 17:12 |
| *** sthillma has quit IRC | 17:12 | |
| fungi | but yeah, i worry this is close to spiralling out of control | 17:12 |
| greghaynes | SpamapS: Oh, so the puppet bifrost module consumes that to generate the inventory? | 17:13 |
| minwang2 | Morning folks, this patch needs another +2 https://review.openstack.org/#/c/211319/, can someone help to take a look at ? | 17:13 |
| greghaynes | SpamapS: crinkle says its a static file... | 17:13 |
| SpamapS | given that Linux will preemptively swap things that aren't doing anything.. any increase in swap is usually a bad sign. | 17:13 |
| clarkb | nope abiword is small resident memory | 17:13 |
| *** dizquierdo has joined #openstack-infra | 17:13 | |
| fungi | also when i killed the abiword process, another spawned to take its place and that one's eating all the cpu now | 17:13 |
| SpamapS | what is abiword doing? | 17:13 |
| clarkb | SpamapS: it does the pdf conversions | 17:13 |
| SpamapS | greghaynes: the puppet bifrost module would consume the erb + hiera to make the inventory json. | 17:14 |
| crinkle | greghaynes: so it could be turned into a template to pass passwords in from hiera | 17:14 |
| SpamapS | clarkb: ow. | 17:14 |
| fungi | yeah, every time i kill abiword, we get a new one | 17:14 |
| *** sputnik13 has joined #openstack-infra | 17:14 | |
| *** dims has quit IRC | 17:14 | |
| greghaynes | got it | 17:14 |
| SpamapS | That seems like something that would be good to make into an async background worker that can be disabled. | 17:14 |
| fungi | SpamapS: i can think of a very easy way to disable it. after all, it's forking /usr/bin/abiword... | 17:14 |
| *** bswartz has quit IRC | 17:15 | |
| greghaynes | fungi: You could even have cp a pdf into place that saays "please stop using the convert to pdf feature" | 17:15 |
| greghaynes | er, have it cp | 17:15 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: WIP: Addition of shade unregister_machine timeout https://review.openstack.org/214285 | 17:15 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: WIP: unregister_machine blocking logic https://review.openstack.org/214286 | 17:15 |
| fungi | greghaynes: hah, good one | 17:15 |
| SpamapS | greghaynes: you can try it.. run 'python tools/make-infracloud-hiera.py inventory-full.json hiera.yaml > inventory.erb | 17:15 |
| *** sthillma has joined #openstack-infra | 17:16 | |
| *** erikwilson has joined #openstack-infra | 17:16 | |
| *** erikwilson is now known as erikwilson_ | 17:16 | |
| *** kiran-r has quit IRC | 17:16 | |
| clarkb | dougwig: so in this job for nodepool tesitng against devstack I should disable the vpn service yes? | 17:16 |
| openstackgerrit | Kurt Taylor proposed openstack-infra/infra-specs: Host an initial CI monitoring dashboard service https://review.openstack.org/194437 | 17:16 |
| *** pc_m has quit IRC | 17:16 | |
| clarkb | dougwig: what should the default devstack runs a neutron service list look like? | 17:17 |
| *** claudiub has joined #openstack-infra | 17:17 | |
| *** pc_m has joined #openstack-infra | 17:17 | |
| SpamapS | I'm guessing if we really wanted we could point our scaling guns at etherpad and make it horizontally scale a lot of the stuff it does. But it sounds like memory makes it perform better.. so .. moar? | 17:17 |
| *** yamahata has quit IRC | 17:17 | |
| *** ayoung has quit IRC | 17:17 | |
| clarkb | SpamapS: you can't actually | 17:17 |
| fungi | SpamapS: no clue how you horizontally scale a single-process node app | 17:18 |
| clarkb | SpamapS: you have to make sure each pad is served by the same server | 17:18 |
| SpamapS | clarkb: nneeevverrr say that to an engineer.. ;) | 17:18 |
| clarkb | SpamapS: the problem is that at these summits you get a single pad that is trouble | 17:18 |
| SpamapS | shard each pad | 17:18 |
| *** bswartz has joined #openstack-infra | 17:18 | |
| clarkb | SpamapS: but the value there is almost nil | 17:18 |
| greghaynes | it has some pretty serious design problems | 17:18 |
| clarkb | for our use case | 17:18 |
| greghaynes | we should really switch to infinited based editing | 17:18 |
| clarkb | our use case is one large pad open | 17:18 |
| greghaynes | but I am biased | 17:18 |
| clarkb | you can't scale that horizontally | 17:18 |
| greghaynes | er, infinoted | 17:18 |
| fungi | in theory we could shard by url, yes, but there are multiple url forms for each pad so we'd need to match or disable some | 17:18 |
| greghaynes | Yes, although we seem to hit our issues by overloading a single pad :( | 17:19 |
| clarkb | greghaynes: right thats my point | 17:19 |
| clarkb | scaling horizontally is useless for our use case | 17:19 |
| fungi | also the etherpad query api would need a proxy of some sort to continue working | 17:19 |
| clarkb | our use case requires scaling up | 17:19 |
| *** pabelanger has joined #openstack-infra | 17:19 | |
| *** dims_ has quit IRC | 17:20 | |
| *** sputnik13 has quit IRC | 17:20 | |
| greghaynes | *caugh*gobby*caugh* | 17:20 |
| clarkb | or a different service | 17:20 |
| clarkb | ya that | 17:20 |
| SpamapS | Right, so break it up into users. Shard each user's text. Use a DLM to serialize updates to a single area so all editors see consistent views while non-editors lag. | 17:20 |
| *** boris-42 has quit IRC | 17:20 | |
| greghaynes | Thers a lib for that! its called infinote :) | 17:20 |
| *** dims has joined #openstack-infra | 17:20 | |
| *** jeadr has quit IRC | 17:20 | |
| clarkb | SpamapS: when you want to rewrite a node.js app let me know | 17:20 |
| *** dims has quit IRC | 17:20 | |
| fungi | yep, at this point we're effectively discussing an etherpad rewrite/replacement | 17:20 |
| SpamapS | clarkb: moar memory instead.. kthxbai ;) | 17:21 |
| clarkb | SpamapS: in the mean time we are restricted to what we can do which is scale up | 17:21 |
| *** dims has joined #openstack-infra | 17:21 | |
| SpamapS | fungi: in Rust? | 17:21 |
| SpamapS | pllzzzz | 17:21 |
| *** jeadr has joined #openstack-infra | 17:21 | |
| fungi | sure, why not | 17:21 |
| pabelanger | clarkb: fungi: mordred: When you have some spare cycles, I'd be interested in feedback on: https://review.openstack.org/#/c/211703/ Wouldn't mind the help to close that out this week. Currently a blocker for fedora22 dibs | 17:21 |
| SpamapS | Sorry my caffeine is kicking in late today. No more yaks to shave.. time to see if I can make this inventory thingy work | 17:21 |
| *** shashank_hegde has quit IRC | 17:21 | |
| *** luqas has quit IRC | 17:22 | |
| *** geoffarnold has joined #openstack-infra | 17:22 | |
| clarkb | dougwig: currently we have q-agt,q-dhcp,q-fwaas,q-l3,q-lbaas,q-meta,q-metering,q-svc,q-vpn,quantum I think I can remove fwaas and vpn, if I do remove fwaas does that break security groups? | 17:23 |
| SpamapS | zomg | 17:23 |
| SpamapS | https://lists.debian.org/debian-devel-announce/2015/08/msg00006.html | 17:23 |
| *** pblaho has quit IRC | 17:23 | |
| SpamapS | The apt in experimental also provides a fix for a long standing race-condition | 17:23 |
| SpamapS | that sometimes, when the archive is getting mirrored users may experience a | 17:23 |
| SpamapS | "hashsum mismatch" error. | 17:23 |
| *** wshao has joined #openstack-infra | 17:24 | |
| clarkb | fungi: before the summit we should 1) sync up etherpad-dev and etherpad versions so that we are not running our local patched version 2) double (or more) the ram available to node on the prod service | 17:24 |
| greghaynes | oh wow | 17:24 |
| *** HeOS has joined #openstack-infra | 17:24 | |
| SpamapS | greghaynes: now apt can really die | 17:24 |
| clarkb | SpamapS: how does it do that? the hash mismatch is a broken mirror | 17:24 |
| *** erikwilson_ has quit IRC | 17:24 | |
| SpamapS | long live docker! ;) | 17:24 |
| SpamapS | clarkb: the hash mismatch is non-atomically-updated metadata files | 17:25 |
| *** erikwilson_ has joined #openstack-infra | 17:25 | |
| clarkb | SpamapS: right the package itself is updated before or after the file wiht the hash | 17:25 |
| SpamapS | clarkb: next sentence, the archive will produce files named by their hash, instead of a static name. | 17:25 |
| clarkb | SpamapS: so not sure how my local apt client works around that | 17:25 |
| *** yamamoto has quit IRC | 17:25 | |
| clarkb | ah so it requires you to build new mirrors | 17:25 |
| SpamapS | clarkb: so the metadata says "Packages hash=12345" and the file is Packages.12345 | 17:25 |
| nigelb | fungi: I hear you guys had a mailman incident too. | 17:25 |
| SpamapS | clarkb: yes because the mirror format is broken. :) | 17:25 |
| SpamapS | archive specifically, not mirror. | 17:26 |
| SpamapS | the mirrors are just doing what they're told | 17:26 |
| clarkb | right | 17:26 |
| nigelb | fungi: We had one too. Is it possible to compare notes? | 17:26 |
| *** dizquierdo has quit IRC | 17:26 | |
| SpamapS | So you do need apt-mirror to DTRT and download all the .$hash files before the larger metadata that refers to them. | 17:26 |
| *** achanda has joined #openstack-infra | 17:27 | |
| SpamapS | anyway, this is good news... apt has had this problem for so long it's embarrassing. | 17:27 |
| sinval | hi guys, can i have some +2 on this https://review.openstack.org/#/c/214170/, simple fix, 1 line change, thanks | 17:28 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Remove q-vpn from service list for dsvm-nodepool https://review.openstack.org/214288 | 17:29 |
| clarkb | greghaynes: ^ I think that will get nodepool job running again | 17:29 |
| greghaynes | fungi: clarkb etherpad no workey | 17:29 |
| sc68cal | clarkb: reads scrollbaack about fwaas | 17:30 |
| greghaynes | just says loading... | 17:30 |
| sc68cal | clarkb: fwaas doesn't handle the security group API | 17:30 |
| *** angdraug has joined #openstack-infra | 17:30 | |
| clarkb | sc68cal: ok, any opinion on whether or not it should be present by default? looks like it is currently in the default list of service | 17:30 |
| *** otter768 has joined #openstack-infra | 17:31 | |
| pc_m | clarkb: Can you point me to a failing dsvm-nodepool review? | 17:31 |
| clarkb | pc_m: https://jenkins06.openstack.org/job/gate-dsvm-nodepool-nv/14/console | 17:31 |
| *** SumitNaiksatam has joined #openstack-infra | 17:31 | |
| pc_m | clarkb: thanks! | 17:31 |
| greghaynes | oh, now it loaded | 17:32 |
| clarkb | greghaynes: the srevice did not restart | 17:32 |
| clarkb | greghaynes: any chance it was local network blip? | 17:32 |
| greghaynes | hrm, I was hard refreshing and ircing just fine | 17:32 |
| *** flepied has quit IRC | 17:32 | |
| *** sergmelikyan has joined #openstack-infra | 17:32 | |
| sc68cal | clarkb: it seems like the kitchen sink is being loaded by default | 17:32 |
| greghaynes | and the presenter was seeing it to so I felt bad | 17:32 |
| *** moravec has quit IRC | 17:32 | |
| pc_m | clarkb: Is there as way to see the devstack log from that? | 17:32 |
| clarkb | pc_m: http://logs.openstack.org/54/211754/4/check/gate-dsvm-nodepool-nv/7f5d792/logs/ | 17:33 |
| *** sputnik13 has joined #openstack-infra | 17:33 | |
| clarkb | sc68cal: yes I think that was requested by neutron at one point | 17:34 |
| clarkb | sc68cal: yall wanted to have all the things enabled everywhere | 17:34 |
| anteaya | I also am seeing etherpad reconnections | 17:35 |
| *** otter768 has quit IRC | 17:35 | |
| dougwig | clarkb: fwaas != security groups (yes, my head just exploded too), so you won't break sg's. | 17:35 |
| sc68cal | clarkb: understood - however I think at least the jobs I've started to write for the gate I've been tryint to explicitly set the ENABLED_SERVICES | 17:36 |
| clarkb | anteaya: greghaynes looks like a disconnect just happened, it stopped logging at about 1733 | 17:37 |
| clarkb | then started logging a bunch of LEAVEs at 1736 | 17:37 |
| greghaynes | \O/ | 17:37 |
| anteaya | clarkb: thanks and :( | 17:37 |
| clarkb | my guess is lolnode | 17:37 |
| anteaya | ? | 17:37 |
| clarkb | things timeout, everyone has a sad and its all worse because overloading the host | 17:37 |
| greghaynes | clarkb: yea, is there anything in syslog? | 17:37 |
| * anteaya is out of the loop | 17:37 | |
| greghaynes | about not enough sockets/etc | 17:37 |
| clarkb | sc68cal: sure I am just saying historically the default is kitchen sink forthat reason | 17:38 |
| *** romcheg has joined #openstack-infra | 17:38 | |
| *** balajin_ has joined #openstack-infra | 17:38 | |
| *** pvaneck has joined #openstack-infra | 17:38 | |
| clarkb | greghaynes: nope just a handful of checksum fails | 17:38 |
| greghaynes | awesome | 17:38 |
| anteaya | clarkb: ops meetup has service temporary unavailable on the big screen | 17:38 |
| anteaya | :( | 17:38 |
| jeblair | abiword is running | 17:38 |
| clarkb | oh! it corresponds to when puppet ran | 17:39 |
| dougwig | clarkb: q-svc q-dhcp q-l3 q-meta q-metering q-lbaas, with q-lbaas soon to be gone. | 17:39 |
| clarkb | fun | 17:39 |
| jeblair | killing abiword | 17:39 |
| greghaynes | haha | 17:39 |
| *** moravec has joined #openstack-infra | 17:39 | |
| clarkb | jeblair: it will just start a new one | 17:39 |
| reed | wait... abiword? | 17:39 |
| sc68cal | clarkb: so, to ensure fwaas continues to be tested, would we need to add to ENABLED_SERVICES in the job definitions for Neutron? sort of like how I did with the linux bridge job? | 17:39 |
| anteaya | sc68cal: can we put a hold on fwaas until we get etherpad back | 17:39 |
| jeblair | clarkb: yeah, but not one that's consuming all resources | 17:39 |
| anteaya | sc68cal: please? | 17:40 |
| *** romcheg1 has quit IRC | 17:40 | |
| jeblair | node is using 70% of ram | 17:40 |
| clarkb | sc68cal: ya | 17:40 |
| jeblair | has anyone restarted it? | 17:40 |
| clarkb | jeblair: we have not restarted it yet | 17:40 |
| jeblair | restarting now | 17:40 |
| clarkb | ok | 17:40 |
| clarkb | I will disable puppet too | 17:40 |
| dougwig | sc68cal: let's talk offline, but in general what needs to happen is to make new fwaas specific jobs run in neutron-fwaas and neutron, and remove fwaas from all other jobs everywhere. | 17:41 |
| dougwig | sc68cal: and yes, you'd enable specifically in those jobs. | 17:41 |
| sc68cal | ok - I'm +1000 on that plan - i'll be quiet now per anteaya | 17:41 |
| jeblair | done | 17:41 |
| anteaya | yay thank you | 17:42 |
| *** regXboi has joined #openstack-infra | 17:42 | |
| anteaya | sc68cal: etherpad is back | 17:42 |
| anteaya | do carry on | 17:42 |
| *** austin81 has left #openstack-infra | 17:42 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Remove unneeded neutron services in dsvm-neutron https://review.openstack.org/214288 | 17:42 |
| *** ybathia has joined #openstack-infra | 17:42 | |
| clarkb | puppet is disabled too | 17:42 |
| clarkb | I think what probably happened was puppet started which made node swap | 17:43 |
| clarkb | then by the time puppet is back and node is swapped back everyone has timed out | 17:43 |
| *** smarcet has quit IRC | 17:43 | |
| *** mfisch has joined #openstack-infra | 17:43 | |
| *** mfisch has left #openstack-infra | 17:43 | |
| *** glauco_ has quit IRC | 17:43 | |
| *** sthillma has quit IRC | 17:43 | |
| *** sharikapongubala has joined #openstack-infra | 17:44 | |
| pleia2 | fungi: pointed nigelb your way, he saw a mailman attack elsewhere last night too, seems it was A Thing | 17:44 |
| clarkb | reed: abiword is how etherpad does pad exports | 17:44 |
| clarkb | reed: we may be able to disable that feature but people seem to like it so probably better to scale up a bit to handle memory needs | 17:44 |
| nigelb | fungi: so I ended up running this with mailman's withlist http://dpaste.com/2E0EXMX and cleared the queue. I'm writing something up to monitor POST requests in mailman's access log to catch this in the future if a different email address is used. | 17:45 |
| *** geoffarnold has quit IRC | 17:45 | |
| *** yamahata has joined #openstack-infra | 17:45 | |
| jeblair | clarkb: i think there's something buggy about that though; it can get stuck forever... :/ | 17:46 |
| *** changbl has joined #openstack-infra | 17:46 | |
| anteaya | clarkb: so someone did an abiword export and that used up too much cpu and etherpad went south? | 17:46 |
| anteaya | or jeblair | 17:46 |
| rcarrillocruz | mordred: heya | 17:46 |
| nigelb | fungi: They've definitely found a nice vulnerability in mailman. I don't want to ban using + to filter email addresses, but at the same time, might need to do something to figure out a permanent solution. let me know if you folks think of something better! :) | 17:46 |
| clarkb | jeblair: would not surprise me | 17:46 |
| * nigelb heads to bed | 17:46 | |
| clarkb | anteaya: I think it was more node was using all the memories then whenever puppet run it swapped out and the service temporarily became unuseable | 17:46 |
| clarkb | because the last mass disconnect corresponded to when puppet ran according to syslog | 17:46 |
| pleia2 | nigelb: night! thanks | 17:46 |
| anteaya | etherpad errored again :( | 17:47 |
| jeblair | also backups are running | 17:47 |
| anteaya | clarkb: ah okay thanks | 17:47 |
| anteaya | ah and backups | 17:47 |
| anteaya | hmmmm | 17:47 |
| nigelb | pleia2: <3 Hopefully it helps. | 17:47 |
| anteaya | hey pleia2 are you at ops meetup? | 17:47 |
| *** geoffarnold has joined #openstack-infra | 17:47 | |
| jeblair | anteaya: she's sick | 17:47 |
| pleia2 | anteaya: no, post travel upset stomach, I plan on joining you all tomorrow | 17:47 |
| clarkb | anteaya: I am not seeing mass LEAVEs logged again | 17:47 |
| anteaya | oh no :( | 17:48 |
| clarkb | it looks healthy based on the logs | 17:48 |
| pleia2 | anteaya: it happens :) rehydrating now | 17:48 |
| anteaya | clarkb: okay might have been a temp blip | 17:48 |
| anteaya | pleia2: yes drink water and sleep | 17:48 |
| pleia2 | anteaya: nods | 17:48 |
| clarkb | oh here are a bunch of ENTERs maybe I missed the LEAVEs | 17:48 |
| anteaya | and looking forward to seeing you tomorrow if that works | 17:48 |
| pleia2 | anteaya: me too! :) | 17:49 |
| anteaya | :( | 17:49 |
| anteaya | sorry that was :) | 17:49 |
| anteaya | and sleep well with cats | 17:49 |
| anteaya | and much tea | 17:49 |
| *** ihrachyshka has joined #openstack-infra | 17:50 | |
| *** hyakuhei has quit IRC | 17:50 | |
| *** wshao has quit IRC | 17:51 | |
| *** ociuhandu has quit IRC | 17:52 | |
| *** dkranz has quit IRC | 17:53 | |
| anteaya | I am 503 on etherpad personally | 17:54 |
| *** sridhar_ram has joined #openstack-infra | 17:54 | |
| reed | back for me | 17:54 |
| reed | but suffering a lot | 17:55 |
| asselin_ | mordred, Shrews clarkb I found the bug: you need to use projectId instaed of tenantId: curl -g -i -X POST http://10.50.218.1:5000/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"passwordCredentials": {"username": "admin", "password": "sanitized"}, "projectId": "admin"}}' | 17:55 |
| clayton | yeap, still having issues here | 17:55 |
| jeblair | ftr, etherpad can not scale horizontally (state is in-memory in a node.js process) | 17:56 |
| *** hdd has quit IRC | 17:56 | |
| clarkb | that load average is pretty impressive for etherpad | 17:56 |
| clarkb | asselin_: fun, oscc just hands that off to the keystoneclient's generic client class | 17:58 |
| clarkb | asselin_: and I thought it was uspposed to do the correct thing, so either a bug in the client or keystone | 17:59 |
| *** gordc has quit IRC | 17:59 | |
| *** ajmiller has joined #openstack-infra | 18:00 | |
| purp | Yeah, etherpad needs some serious scaling love. | 18:01 |
| purp | (from etherpad devs, not y'all) | 18:01 |
| asselin_ | clarkb, must be the client. Keystone juno's been working for over a year now. I'll look into that | 18:01 |
| *** fedora has joined #openstack-infra | 18:01 | |
| *** tiswanso has quit IRC | 18:02 | |
| * david-lyle wishes horizontally started with not horizon | 18:02 | |
| anteaya | hey david-lyle! | 18:03 |
| anteaya | :) | 18:03 |
| *** ayoung has joined #openstack-infra | 18:03 | |
| *** tiswanso has joined #openstack-infra | 18:03 | |
| clarkb | asselin_: there was a recent change in oscc to make the keystone v2 handler just use the generic handler | 18:04 |
| david-lyle | anteaya: :) | 18:04 |
| *** ftersin has left #openstack-infra | 18:04 | |
| clarkb | asselin_: that may be the actual genesis of the bug | 18:04 |
| pelix | zaro: guess lots of small files aren't going to be a problem with the python-jenkins tests | 18:04 |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Add a non-voting py34 check job to Horizon https://review.openstack.org/214292 | 18:04 |
| *** serverascode has joined #openstack-infra | 18:05 | |
| *** abregman has joined #openstack-infra | 18:05 | |
| *** abregman has quit IRC | 18:05 | |
| *** chlong has quit IRC | 18:06 | |
| wznoinsk | asselin_: re the 401 error in nodepool you see, would that be an older version of the client (openstack or keystone) that's causing it? | 18:06 |
| *** abregman has joined #openstack-infra | 18:06 | |
| *** shashank_hegde has joined #openstack-infra | 18:06 | |
| *** fedora has quit IRC | 18:06 | |
| wznoinsk | I think I've just reiterated what clarkb said, sry | 18:06 |
| *** gordc has joined #openstack-infra | 18:07 | |
| jeblair | Shrews, mordred: it looks like you need to do cloud.nova_client.flavors.list() in order to get the flavor for create_server... | 18:07 |
| *** glauco_ has joined #openstack-infra | 18:07 | |
| *** amotoki has quit IRC | 18:07 | |
| jeblair | but the docs say that poking at the underlying client objects is frowned upon | 18:07 |
| zaro | pelix: does that mean good idea? | 18:08 |
| jeblair | but that's like the basic example of what shade is supposed to do simply | 18:08 |
| *** gyee has joined #openstack-infra | 18:08 | |
| jeblair | greghaynes: ^? | 18:08 |
| jeblair | am i just missing the simple way to create a server with one method? | 18:08 |
| greghaynes | jeblair: Yes, so we currently poke the underlying clients for everything but image uploading so its sort of 'accepted for infra while shade gets to where it should be' | 18:09 |
| *** chlong has joined #openstack-infra | 18:09 | |
| asselin_ | clarkb, wznoinsk akraminsky python-opentstack client isn't installed. os that oscc? http://paste.openstack.org/show/420815/ | 18:09 |
| clarkb | I have to pop out and find food now and prep for another 100F day | 18:09 |
| greghaynes | I dont actually know of a method to do that... | 18:09 |
| clarkb | back in a bit | 18:09 |
| clarkb | asselin_: oscc is openstack client config | 18:09 |
| jeblair | :/ | 18:09 |
| clarkb | its a shade dep | 18:09 |
| anteaya | clarkb: happy shopping/eating | 18:10 |
| asselin_ | clarkb, thanks will look into that | 18:10 |
| greghaynes | I am not extremely versed in shade methods though, so probably want to see what Shrews or mordred say | 18:10 |
| wznoinsk | asselin_: check yum/apt for python-openstack* | 18:10 |
| Shrews | wait... what? | 18:10 |
| Shrews | jeblair: what do you mean? i don't understand what the problem is | 18:10 |
| jeblair | Shrews: i'm trying to write example code that creates a server | 18:10 |
| jeblair | Shrews: but i don't see how to do that without touching cloud.nova_client | 18:11 |
| jeblair | to get flavors and images | 18:11 |
| *** fedora has joined #openstack-infra | 18:11 | |
| *** openstackgerrit_ has quit IRC | 18:11 | |
| Shrews | jeblair: cloud.list_flavors() | 18:11 |
| Shrews | jeblair: cloud.get_flavor('my_favorite_name') | 18:11 |
| fungi | nigelb: i think someone with a bot army was using all the mailman servers they could find to use in a reflection attack. for us it lasted from 09:27 utc yesterday to 08:45 utc today | 18:11 |
| Shrews | jeblair: cloud.list_images() | 18:12 |
| jeblair | Shrews: ah, the create_server func test doesn't use that | 18:12 |
| Shrews | jeblair: no. you HAVE to tell it what flavor you want | 18:12 |
| Shrews | and image | 18:12 |
| *** salv-orlando has quit IRC | 18:12 | |
| Shrews | we can't guess it | 18:12 |
| fungi | nigelb: the + extension is going to be mta-specific, so probably was used in this case because gmail happens to implement it | 18:12 |
| jeblair | Shrews: that i understand :) | 18:12 |
| jeblair | Shrews: i was objecting to having to touch the nova client object | 18:12 |
| Shrews | jeblair: yes, i object to that too! :) | 18:12 |
| Shrews | you shouldn't ever need to. if you do, we've missed something | 18:13 |
| jeblair | Shrews: yeah, that's why i came to you :) | 18:13 |
| fungi | and i guess etherpad died/restarted while i stepped away for a few minutes | 18:13 |
| jeblair | Shrews: is there a get_image method? | 18:13 |
| jeblair | fungi: i restarted it to try to make it less terrible | 18:13 |
| Shrews | jeblair: yes | 18:13 |
| Shrews | jeblair: so, most resources have a get/list/search method | 18:14 |
| jlvillal | Ironic gate might be broken: https://review.openstack.org/#/c/209605/ https://review.openstack.org/#/c/213011/ are failing the gate-tempest-dsvm-ironic-pxe_ssh-postgres | 18:14 |
| Shrews | get_blah(), list_blahs(), search_blahs() | 18:14 |
| jeblair | Shrews: how can i use get_image to say "get me trusty"? | 18:14 |
| asselin_ | wznoinsk, none | 18:15 |
| jlvillal | The ironic failures seemed to have started sometime in the last 10-24 hours or so. | 18:15 |
| Shrews | jeblair: you give it either the image name or the image id | 18:15 |
| Shrews | jeblair: name, if it is a unique name | 18:15 |
| Shrews | jeblair: image = get_image('Ubuntu Trusty') | 18:15 |
| Shrews | or whatever | 18:16 |
| jeblair | server = cloud.create_server(name='test_create_server', | 18:16 |
| jeblair | image=cloud.get_image('Debian Wheezy'), | 18:16 |
| jeblair | flavor=cloud.get_flavor_by_ram(8192)) | 18:16 |
| *** pcrews has joined #openstack-infra | 18:16 | |
| jeblair | Shrews: that look reasonable? | 18:16 |
| *** geoffarnold has quit IRC | 18:17 | |
| Shrews | jeblair: yeah, but you can also get flavors by name or id | 18:17 |
| *** moravec has quit IRC | 18:17 | |
| Shrews | if you want | 18:17 |
| *** geoffarnold has joined #openstack-infra | 18:18 | |
| jeblair | oh so: get_flavor('8G') | 18:18 |
| jeblair | ? | 18:18 |
| *** jeadr has quit IRC | 18:18 | |
| *** liam__ has joined #openstack-infra | 18:19 | |
| jeblair | Shrews: i'm preparing a lightning talk on shade for the ops meetup | 18:19 |
| *** jeadr has joined #openstack-infra | 18:19 | |
| Shrews | jeblair: 8G is not a name or id :) you have to use the _by_ram one for ram based searching (it's an older method) | 18:19 |
| *** HeOS has quit IRC | 18:19 | |
| jeblair | Shrews: well, it is if you have a flavor named '8G', right? | 18:19 |
| pelix | zaro: unless you think we need to avoid it, I'm happy with it, I was just concerned previously that 'that' would be a concern ;) | 18:20 |
| Shrews | jeblair: if you are a terrible operator, yes :) | 18:20 |
| jeblair | Shrews: maybe a better example would be "medium" ? | 18:20 |
| *** dannywilson has quit IRC | 18:20 | |
| jeblair | Shrews: yes, using terrible operators as examples is a good idea in this instance. :) | 18:20 |
| Shrews | jeblair: reed actually has a documented example on docs.o.o somewhere | 18:20 |
| jeblair | or i guess i could use 'vanilla' | 18:21 |
| asselin_ | tim_o, you have a question? | 18:21 |
| * fungi votes in favor of "vanilla cloud" | 18:21 | |
| reed | Shrews, my patch merged but I can't see it on docs.o.o for some reason, I haven't investigated | 18:21 |
| reed | it's on the git repo though... | 18:22 |
| *** megm has quit IRC | 18:22 | |
| *** ivar-lazzaro has quit IRC | 18:22 | |
| Shrews | reed: can you share that link with jeblair? | 18:22 |
| fungi | reed: depending on the repo, it may be set to only update on release | 18:22 |
| reed | https://git.openstack.org/cgit/openstack/api-site/tree/firstapp/source/getting_started.rst | 18:22 |
| reed | and the sample code is https://git.openstack.org/cgit/openstack/api-site/tree/firstapp/samples/shade | 18:23 |
| jeblair | reed: thx! | 18:24 |
| reed | sure | 18:24 |
| fungi | reed: so https://review.openstack.org/203796 was the change? | 18:24 |
| reed | fungi, that's it | 18:24 |
| *** notnownikki has quit IRC | 18:25 | |
| fungi | i don't see another commit merged to that repo around the same time, so unlikely to be our publication race friend | 18:25 |
| fungi | more likely it was queued and then we restarted zuul last week and lost the contents of the post pipeline | 18:25 |
| *** dannywilson has joined #openstack-infra | 18:25 | |
| wznoinsk | asselin_: did you check the versions of pip packages in the venv as well? | 18:25 |
| reed | it's possible that the document is not considered complete because it contains only the 'getting started' page | 18:25 |
| *** fifieldt has joined #openstack-infra | 18:25 | |
| fifieldt | hi team | 18:25 |
| anteaya | what is the current etherpad status? | 18:26 |
| reed | I haven't touched 'scaling' and the other ones | 18:26 |
| *** yamamoto has joined #openstack-infra | 18:26 | |
| anteaya | we are still seeing reconnects and errors | 18:26 |
| jlvillal | Any known issues that would have caused the Ironic check gate to break on the gate-tempest-dsvm-ironic-pxe_ssh-postgres job? | 18:26 |
| *** abregman is now known as abregman|afk | 18:26 | |
| *** moravec has joined #openstack-infra | 18:26 | |
| anteaya | fifieldt: did I steal your thunder? | 18:26 |
| fifieldt | not at all - it was quite convenient :) | 18:26 |
| anteaya | fifieldt: timing \o/ | 18:26 |
| fungi | anteaya: looks like etherpad's been running continuously since 17:40 utc, so about 50 minutes | 18:26 |
| reed | fifieldt: we were wondering why the shade getting started bit is not listed on docs.o.o yet... is it because the tutorial is not complete yet? | 18:27 |
| anteaya | we keep having very frequent reconnects | 18:27 |
| anteaya | and many errors | 18:27 |
| fifieldt | yeah, it's bad enough I'm thinking about running a local etherpad server | 18:27 |
| jeblair | anteaya: load average of something like 140. i'm not aware of any obvious issues (unlike earlier) | 18:27 |
| fifieldt | reed, that is expected | 18:27 |
| fifieldt | it won't be published until it's complete | 18:27 |
| anteaya | and 503 | 18:27 |
| anteaya | okay fair enough | 18:27 |
| reed | fifieldt, i imagined that... ok, thanks | 18:27 |
| fungi | anteaya: yeah, the tcp open stats for the server have skyrocketed. before the restart they were around 40, after they're around 250 | 18:28 |
| *** geoffarnold has quit IRC | 18:28 | |
| fifieldt | I think there's some draft build state | 18:28 |
| anteaya | what should I be doing diffently so that I don't keep getting 302 | 18:28 |
| fifieldt | 250 is about right for this event | 18:28 |
| jlvillal | And who do we contact about someone spamming the Ironic channel with 'Alba naked pics'? | 18:28 |
| clarkb | jlvillal: any of your channel ops accrding to chanser | 18:28 |
| anteaya | jlvillal: any channel op many of whom are in here | 18:28 |
| jlvillal | Thanks, devananda became op :) | 18:29 |
| fungi | memory usage is much better on the server now. "system" cpu has gotten a fair bit worse though | 18:29 |
| *** liam__ has quit IRC | 18:29 | |
| fungi | looks like a lot of i/o wait | 18:29 |
| *** flaviosr has joined #openstack-infra | 18:29 | |
| *** jeadr has quit IRC | 18:29 | |
| jeblair | fungi: backups are running | 18:29 |
| fungi | oh | 18:29 |
| jeblair | could kill them see if that improves it | 18:30 |
| reed | jeblair, the draft build: http://docs-draft.openstack.org/96/203796/19/check/gate-api-site-tox-doc-publish-checkbuild/3f4fb00//publish-docs/draft/firstapp-shade/index.html | 18:30 |
| *** ivar-lazzaro has joined #openstack-infra | 18:30 | |
| *** hyakuhei has joined #openstack-infra | 18:30 | |
| *** jeadr has joined #openstack-infra | 18:30 | |
| fungi | huh... bup start time was 05:37 and it's still going? | 18:30 |
| *** pcrews has quit IRC | 18:30 | |
| fungi | 13 hours later | 18:30 |
| fungi | if i'm reading that correctly | 18:30 |
| *** yamamoto has quit IRC | 18:30 | |
| jeblair | i killed it | 18:30 |
| anteaya | thanks | 18:31 |
| *** gyee has quit IRC | 18:31 | |
| jeblair | i wonder if it took a long time to make the db dump? | 18:31 |
| fungi | yeah, definitely no longer running ;) | 18:31 |
| anteaya | lets see if that makes an improvement | 18:31 |
| fifieldt | thanks guys | 18:31 |
| * fifieldt crosses fingers | 18:31 | |
| *** skiafa_ has joined #openstack-infra | 18:31 | |
| fungi | jeblair: i don't think it was the db dump... it was a tar|bup process started at 05:37 | 18:31 |
| anteaya | etherpads are rendering! | 18:32 |
| anteaya | yay | 18:32 |
| *** gyee has joined #openstack-infra | 18:32 | |
| jeblair | fungi: yeah, that's what i killed -- it was crawling the fs and streaming some of clarkb's old db dumps | 18:32 |
| fifieldt | reed, there is some magic we can put into the tox.ini to make it build a shade constatntly at docs.o.o/draft | 18:32 |
| fifieldt | I think | 18:32 |
| fifieldt | but I don't know the exact syntax | 18:32 |
| clarkb | jeblair: oh! whoops | 18:32 |
| jeblair | fungi: but my suggestion was that maybe it was still running because of earlier trouble/slowness with making the current db dump? | 18:32 |
| jeblair | -rw-r--r-- 1 root root 0 Aug 18 07:07 etherpad-lite.sql.gz | 18:32 |
| jeblair | -rw-r--r-- 1 root root 407230071 Aug 18 00:03 etherpad-lite.sql.gz.1 | 18:32 |
| jeblair | that looks weird. | 18:33 |
| clarkb | jeblair: thats due to logrotate | 18:33 |
| reed | fifieldt, i'll look into that, thanks... some documentation is better than none :) | 18:33 |
| fifieldt | indeedy | 18:33 |
| *** mtrovo has quit IRC | 18:33 | |
| fungi | jeblair: i've noticed there's some sort of off-by-one on our database dumps. we get a zero-byte most recent backup and an actual most recent backup of .1 with the content | 18:33 |
| *** kgiusti has quit IRC | 18:33 | |
| clarkb | fungi: jeblair yes due to logrotate | 18:33 |
| fungi | haven't dug deeper into it since we still seem to actually be dumping the database on schedule and successfully | 18:34 |
| skiafa_ | hi folks, I was wondering if someone is working on the manage/unmanage volume tests... does anybody know? | 18:34 |
| * fifieldt is still getting reconnects :| | 18:34 | |
| anteaya | fifieldt: :| | 18:34 |
| fungi | clarkb: oh, that would make sense. forgot we're relying on logrotate to do that for them | 18:34 |
| fungi | clarkb: i think there's a logrotate setting to not create the new logfile, we could probably just set that | 18:35 |
| clarkb | fungi: if that exists then definitely | 18:35 |
| fifieldt | things are working a bit better than before though | 18:35 |
| *** nadya has joined #openstack-infra | 18:35 | |
| clarkb | bandersnatch is still snatching things | 18:35 |
| anteaya | small yay? | 18:36 |
| fungi | clarkb: according to man 8 lotrotate it seems to be called "nocreate" | 18:36 |
| jeblair | what uses ossc now -- shade and openstackclient, right? | 18:37 |
| jeblair | dhellmann: ^? | 18:37 |
| clarkb | jeblair: shade definitely does | 18:37 |
| *** hyakuhei has quit IRC | 18:37 | |
| jeblair | oscc | 18:38 |
| jeblair | i always get that wrong | 18:38 |
| jeblair | can we change it? ;) | 18:38 |
| stevemar | jeblair: yes, openstackclient uses oscc | 18:38 |
| jeblair | cool, thx | 18:38 |
| *** sanjayu_ has quit IRC | 18:39 | |
| openstackgerrit | Noorul Islam K M proposed openstack-infra/jenkins-job-builder: Use correct element name in scoverage module https://review.openstack.org/214299 | 18:39 |
| *** nadya has quit IRC | 18:39 | |
| *** hyakuhei has joined #openstack-infra | 18:40 | |
| *** sanjayu has joined #openstack-infra | 18:40 | |
| *** gordc has quit IRC | 18:40 | |
| *** geoffarnold has joined #openstack-infra | 18:40 | |
| *** dkranz has joined #openstack-infra | 18:41 | |
| openstackgerrit | Noorul Islam K M proposed openstack-infra/jenkins-job-builder: Use correct element name in scoverage module https://review.openstack.org/214299 | 18:41 |
| *** puranamr has quit IRC | 18:41 | |
| reed | fifieldt, looks like the publication in /draft/ site at least for manuals is done via a shell script in tools/build-all-rst.sh | 18:42 |
| *** pelix has quit IRC | 18:42 | |
| fifieldt | andreas is the master there :) | 18:42 |
| fifieldt | jaegar that is | 18:42 |
| fifieldt | christian berendt also knows | 18:42 |
| reed | yeah... looks easier to just finish the damn thing :) | 18:43 |
| fungi | pabelanger: for 211294 didn't you say you need to figure out how to make dnf download into the yum/rpm cache location rather than pwd? | 18:43 |
| openstackgerrit | Sahdev Zala proposed openstack-infra/project-config: Add new repository for stand-alone TOSCA Parser https://review.openstack.org/211204 | 18:44 |
| anteaya | still seeing some 302 on the etherpad | 18:45 |
| *** kgiusti has joined #openstack-infra | 18:49 | |
| anteaya | another 302 for fifieldt | 18:50 |
| *** ihrachyshka has quit IRC | 18:50 | |
| *** hyakuhei has quit IRC | 18:50 | |
| *** puranamr has joined #openstack-infra | 18:51 | |
| pabelanger | fungi: right now, it should work the same way yum --downloadonly works | 18:51 |
| pabelanger | fungi: however, I _think_ that just downloads the files into the current directory too | 18:52 |
| *** Sukhdev has joined #openstack-infra | 18:52 | |
| fungi | er... huh | 18:52 |
| fungi | i'll have to test that again but i thought it put them in /var/cache/yum | 18:52 |
| pabelanger | fungi: If I am reading correctly, we also need to setup the --destdir flag for it | 18:55 |
| pabelanger | err... downloaddir? | 18:55 |
| *** ociuhandu has joined #openstack-infra | 18:56 | |
| *** geoffarnold has quit IRC | 18:56 | |
| jeblair | clarkb, fungi, Shrews, mordred, greghaynes: https://etherpad.openstack.org/p/TUFB3vFoNz | 18:56 |
| jeblair | draft lightning talk on shade + oscc for ops meetup | 18:56 |
| *** sdake_ has joined #openstack-infra | 18:56 | |
| *** wshao has joined #openstack-infra | 18:56 | |
| *** tiswanso has quit IRC | 18:57 | |
| jhesketh | Morning | 18:58 |
| Shrews | jeblair: neat. hope it goes well | 18:58 |
| jeblair | i'm imagining the verbal content will be a combination of helpful explanation of shade + rant about deployment differences | 18:58 |
| Shrews | moar rantz | 18:58 |
| jeblair | lunching now | 18:59 |
| SpamapS | we infra meeting-ing? | 18:59 |
| fungi | SpamapS: yep | 18:59 |
| *** sgordon has quit IRC | 18:59 | |
| SpamapS | kk | 18:59 |
| *** angdraug has quit IRC | 18:59 | |
| *** sgordon has joined #openstack-infra | 18:59 | |
| SpamapS | Whats the capacity of pbx.openstack.org btw? Like, if we wanted to have an actual conference call.. could we even swing that? | 18:59 |
| *** sdake has quit IRC | 18:59 | |
| * SpamapS just really sucks at IRC meetings.. :-P | 19:00 | |
| fungi | we've load tested it pretty heavily bot i don't recall the exact numbers | 19:00 |
| fungi | should work for reasonable sized conference calls at the very least | 19:00 |
| *** tiswanso has joined #openstack-infra | 19:00 | |
| *** tjones has quit IRC | 19:00 | |
| *** gyee has quit IRC | 19:00 | |
| jlvillal | FYI: Regarding Ironic gate being broken. Currently pursuing the issue with the neutron team. Appears to be a change made to neutron. | 19:01 |
| fungi | meeting time | 19:01 |
| clarkb | you run vpn service? | 19:01 |
| *** ybathia has quit IRC | 19:02 | |
| *** fawadkhaliq has joined #openstack-infra | 19:02 | |
| *** ybathia has joined #openstack-infra | 19:03 | |
| *** regXboi has quit IRC | 19:03 | |
| *** romcheg has quit IRC | 19:05 | |
| *** hdd has joined #openstack-infra | 19:05 | |
| *** annegentle has joined #openstack-infra | 19:06 | |
| *** austin81 has joined #openstack-infra | 19:06 | |
| *** wshao has quit IRC | 19:07 | |
| *** e0ne has joined #openstack-infra | 19:07 | |
| *** amotoki has joined #openstack-infra | 19:08 | |
| *** ayoung has quit IRC | 19:08 | |
| *** nadya has joined #openstack-infra | 19:09 | |
| *** kiran-r has joined #openstack-infra | 19:10 | |
| *** _kiran_ has joined #openstack-infra | 19:11 | |
| *** ianw has quit IRC | 19:11 | |
| *** romcheg has joined #openstack-infra | 19:12 | |
| *** amotoki has quit IRC | 19:13 | |
| *** ianw has joined #openstack-infra | 19:13 | |
| *** fawadk has joined #openstack-infra | 19:13 | |
| *** kiran-r has quit IRC | 19:13 | |
| *** olaph has joined #openstack-infra | 19:14 | |
| *** esker has joined #openstack-infra | 19:15 | |
| *** w_ has quit IRC | 19:15 | |
| *** fawadkhaliq has quit IRC | 19:16 | |
| *** salv-orl_ has joined #openstack-infra | 19:16 | |
| pc_m | ianw: ping | 19:16 |
| *** sdake_ is now known as sdake | 19:16 | |
| *** w_ has joined #openstack-infra | 19:16 | |
| ianw | pc_m: hi | 19:16 |
| *** shashank_hegde has quit IRC | 19:16 | |
| pc_m | ianw: On https://review.openstack.org/#/c/209887/ I wanted to get some clarification. | 19:16 |
| pc_m | ianw: Are you suggesting that we don't make the gate-hook.sh conditionally set? | 19:16 |
| pc_m | IOW have the job require that the hook be used. | 19:17 |
| *** mmmpork has left #openstack-infra | 19:18 | |
| ianw | pc_m: ATM the job unconditionally calls gate_hook.sh, which isn't committed yet | 19:18 |
| *** mmmpork has joined #openstack-infra | 19:18 | |
| clarkb | just waiting on hpcloud pypi mirror to finish updating at this point | 19:18 |
| pabelanger | fungi: not sure I follow your comments on 186619. Do we not want a devstack-fedora22 node? | 19:18 |
| fungi | pabelanger: i think we want a fedora-22 node | 19:19 |
| ianw | pc_m: I'm of the opinion doing [ -f .../gate_hook.sh ] is a bad idea, because it introduces potential for silent failure | 19:19 |
| fungi | pabelanger: the only difference is the name | 19:19 |
| *** olaph has quit IRC | 19:19 | |
| fungi | pabelanger: not the content | 19:19 |
| pc_m | ianw: You mean doing a test for gate hook is a bad idea? | 19:20 |
| *** HeOS has joined #openstack-infra | 19:20 | |
| Zara | thanks for mentioning the storyboard spec just now. :) | 19:20 |
| pabelanger | fungi: okay, so it will be used for devstack, we are just chaging the naming format | 19:21 |
| fungi | pabelanger: yep. we can run devstack jobs on our ubuntu-trusty nodes already. we cache devstack stuff on them just like is being proposed for the fedora 22 images | 19:21 |
| *** annegentle has quit IRC | 19:21 | |
| pc_m | ianw: Here's my thought process on this... maybe it'll help with what I was trying to do... | 19:21 |
| ianw | pc_m: my feeling is that having gate hooks as optional is a bad idea. anything that has a failure mode of "silently fail" is a bad idea in a CI job | 19:22 |
| *** achanda has quit IRC | 19:22 | |
| pc_m | ianw: First, commit this new template which uses a gate hook. This job would not be use. | 19:22 |
| pabelanger | fungi: okay, updating name | 19:22 |
| pc_m | ianw: Second, commit the neutronclient change that adds the gate hook and enables the test cases. Run the experimental job on that commit. | 19:23 |
| pabelanger | ianw: woah | 19:23 |
| pc_m | ianw: Third, it it all works, then remove the old job that doesn't use the hook. | 19:23 |
| *** olaph has joined #openstack-infra | 19:23 | |
| *** fifieldt has quit IRC | 19:23 | |
| *** w_ has quit IRC | 19:23 | |
| pc_m | ianw: Is that a reasonable approach? | 19:23 |
| ianw | pc_m: so i'd suggest that is the wrong order. as reviewers, we want to see that the project has signed-off on testing approaches before we start adding jobs. and we want to add robust & minimal jobs. so get your changes committed, add the job as experimental or non-voting, and go from there | 19:24 |
| clarkb | pabelanger: also you odn't need the -dib suffix is starting a completely new thing | 19:24 |
| fungi | pabelanger: yeah, we don't want to have separate bare and devstack varieties of nodes any longer, so we're transitioning to naming the new node types after just their distro and version | 19:24 |
| clarkb | pabelanger: we had been using the -dib suffix to differentiate between the existing devstack-trusty and dib devstack-trusty so we can test them with minimal impact | 19:25 |
| pc_m | ianw: OK. Wanted to save an extra commit. | 19:25 |
| *** annegentle has joined #openstack-infra | 19:26 | |
| pc_m | hence the reasoning for the order proposed | 19:26 |
| *** ivar-laz_ has joined #openstack-infra | 19:26 | |
| *** flaviosr has quit IRC | 19:26 | |
| *** doug-fish has quit IRC | 19:26 | |
| *** minwang2 has quit IRC | 19:27 | |
| *** doug-fish has joined #openstack-infra | 19:27 | |
| mmmpork | clarkb: or greghaynes: regarding nodepool, i'm reviewing some of the "testing" documentation in the README and I was wondering, if you don't create the openstack_citest stuff in the DB, can you still run nodepool? | 19:28 |
| *** ivar-lazzaro has quit IRC | 19:28 | |
| clarkb | mmmpork: yes but you can't run the functional tests | 19:28 |
| clarkb | they depend on that user existing and fail otherwise | 19:28 |
| pabelanger | clarkb: fungi: ack. Naming it fedora-22 now | 19:28 |
| mmmpork | i'm just trying to figure out if that section for setting up the mysql stuff can be removed from the README and only live in the testing doc | 19:29 |
| fungi | pabelanger: we're discussing your meeting topic in #openstack-meeting. care to join? | 19:29 |
| *** geoffarnold has joined #openstack-infra | 19:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add Fedora 22 DIB nodes https://review.openstack.org/186619 | 19:30 |
| *** hyakuhei has joined #openstack-infra | 19:31 | |
| *** otter768 has joined #openstack-infra | 19:31 | |
| *** angdraug has joined #openstack-infra | 19:32 | |
| mmmpork | clarkb: thanks, i'm going to remove that section from the README and reference the testing doc instead | 19:33 |
| mmmpork | i'm working on the redundancy issue :) | 19:33 |
| *** otter768 has quit IRC | 19:36 | |
| *** geoffarnold has quit IRC | 19:38 | |
| *** geoffarnold has joined #openstack-infra | 19:38 | |
| *** w_ has joined #openstack-infra | 19:40 | |
| *** fawadk has quit IRC | 19:41 | |
| *** olaph has quit IRC | 19:43 | |
| *** mestery has joined #openstack-infra | 19:44 | |
| *** w_ has quit IRC | 19:44 | |
| *** e0ne has quit IRC | 19:44 | |
| *** pc_m has quit IRC | 19:46 | |
| *** gordc has joined #openstack-infra | 19:50 | |
| *** ayoung has joined #openstack-infra | 19:51 | |
| openstackgerrit | Augustina Ragwitz proposed openstack-infra/nodepool: Overall documentation improvements focused on setup and testing https://review.openstack.org/214319 | 19:52 |
| *** regXboi has joined #openstack-infra | 19:53 | |
| *** ivar-laz_ has quit IRC | 19:55 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:56 | |
| *** annegentle has quit IRC | 19:57 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:57 | |
| *** mrunge has quit IRC | 19:58 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:58 | |
| lifeless | morning | 19:59 |
| lifeless | hyakuhei: pong | 19:59 |
| lifeless | bah | 19:59 |
| lifeless | hyakuhei: sorry | 19:59 |
| lifeless | HenryG: pong | 19:59 |
| openstackgerrit | Kurt Taylor proposed openstack-infra/infra-specs: Host an initial CI monitoring dashboard service https://review.openstack.org/194437 | 20:00 |
| pleia2 | thanks fungi! | 20:00 |
| *** annegentle has joined #openstack-infra | 20:00 | |
| *** bpokorny_ has joined #openstack-infra | 20:00 | |
| *** bpokorny has quit IRC | 20:00 | |
| *** bpokorny_ has quit IRC | 20:00 | |
| *** bpokorny has joined #openstack-infra | 20:01 | |
| *** tonytan4ever has quit IRC | 20:01 | |
| *** sridhar_ram has quit IRC | 20:02 | |
| *** smarcet has joined #openstack-infra | 20:02 | |
| pradk | question on adding new dependencies to requirements.txt .. I added a new dep thats we recently put on pypi, i can manually install it via pip, but seems to fail via jenkins. Is there anything else that needs to be done other than adding it to requirements? | 20:02 |
| clarkb | pradk: you have to add it to openstack/requirements if the project you are adding it to is enforcing those requirements | 20:03 |
| *** sridhar_ram has joined #openstack-infra | 20:03 | |
| HenryG | lifeless: nm, all sorted, thanks | 20:04 |
| pradk | clarkb, thx for the tip will try that | 20:04 |
| clarkb | pradk: the readme there should explain it all in much depth | 20:04 |
| pradk | cool, thx | 20:05 |
| *** hyakuhei has quit IRC | 20:05 | |
| *** mestery has quit IRC | 20:08 | |
| *** tonytan4ever has joined #openstack-infra | 20:08 | |
| *** mestery has joined #openstack-infra | 20:10 | |
| *** msuriar_ is now known as msuriar | 20:11 | |
| *** minwang2 has joined #openstack-infra | 20:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding python3-jobs to test python-oneviewclient project https://review.openstack.org/214170 | 20:14 |
| *** shashank_hegde has joined #openstack-infra | 20:14 | |
| *** abregman|afk is now known as abregman | 20:16 | |
| *** hyakuhei has joined #openstack-infra | 20:17 | |
| *** hyakuhei has quit IRC | 20:17 | |
| *** boris-42 has joined #openstack-infra | 20:19 | |
| clarkb | sc68cal: re https://review.openstack.org/#/c/210647/ should I keep that around longer or has it served its purpose? I know tempest filed some bugs aginst the floating IP stuff but I don't think any work has been done to fix tempest yet | 20:19 |
| *** btully has joined #openstack-infra | 20:19 | |
| clarkb | also pypi mirror in hpcloud is still mirroring | 20:20 |
| *** puranamr has quit IRC | 20:21 | |
| *** achanda has joined #openstack-infra | 20:22 | |
| SpamapS | hrm | 20:22 |
| *** achanda has quit IRC | 20:22 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/requirements: Add jsonpath-rw-ext to global requirements https://review.openstack.org/214331 | 20:22 |
| SpamapS | so why does hiera have to be referenced in site.pp only? | 20:22 |
| nibalizer | can project-config cores review https://review.openstack.org/#/c/208799/ ? Needed to get the beaker testing in place | 20:22 |
| SpamapS | Why can't you just call out to it inside an erb template for instance? | 20:22 |
| nibalizer | SpamapS: it doesn't, just a convention | 20:22 |
| *** achanda has joined #openstack-infra | 20:23 | |
| clarkb | its a sanity thing if you bury hiera lookups deep in your modules you require the use of hiera | 20:23 |
| nibalizer | most sites have hiera calls everywhre | 20:23 |
| clarkb | and you require specific keys | 20:23 |
| clarkb | and its just bad | 20:23 |
| nibalizer | and telling where a value is comming from becomes very hard | 20:23 |
| SpamapS | Hrm | 20:23 |
| clarkb | if instead you provide a clean interface to the module then let your users decide how to feed the data its much nicer | 20:23 |
| nibalizer | especially since the hiera stuff isn't public | 20:23 |
| SpamapS | We need an IPMI password per physical server at this point.. | 20:23 |
| nibalizer | SpamapS: so use hiera | 20:23 |
| nibalizer | hiera can lookup the key 'ipmi password' and you can have hiera search on fqdn and give a different answer based on 'who is asking' | 20:24 |
| SpamapS | so should I just make direct calls to hiera from the erb that generates the inventory that feeds into Bifrost/Ironic? | 20:24 |
| fungi | assuming each server has a node def in site.pp, passing hiera keys in from there is already our way | 20:24 |
| sc68cal | clarkb: up to you, I have hardware coming for doing IPv6 only testing | 20:24 |
| *** bswartz has quit IRC | 20:24 | |
| jeblair | clarkb: i'm in a session about logging; anything i should be aware of? | 20:24 |
| SpamapS | nibalizer: no, this isn't attached to the host it's running on | 20:24 |
| SpamapS | nibalizer: 100 hosts are managed on the two Ironic nodes. | 20:25 |
| clarkb | jeblair: before twins arrived we had talked about possibly making an upstream logstash ruleset that infra and others could reconsume | 20:25 |
| fungi | jeblair: getting operators to converge on logstash grok filters on ways that can make it back upstream to us would be wunderbar | 20:25 |
| clarkb | jeblair: it would be really cool if someone wanted to pick that work back up again | 20:25 |
| clarkb | yup that | 20:25 |
| nibalizer | so what about hiera('ipmi_passwords') and return a list of passwords? or a hash of server: password ? | 20:25 |
| fungi | er, in ways | 20:25 |
| jeblair | clarkb, fungi: any wip on that? | 20:25 |
| clarkb | jeblair: because what has happend is they all took our rules the never contributed any improvements back | 20:25 |
| clarkb | which makes me sad panda reading through all the downstream users of our thing | 20:26 |
| fungi | jeblair: they had some github repo they were collaborating in, but i don't recall the url now | 20:26 |
| fungi | presumably someone in that session has more details | 20:26 |
| jeblair | clarkb: should we say we're going to stick our filters in a standalone repo, consume them from there, and they should show up and pitch in? :) | 20:26 |
| fungi | i can get behind that. seems simple to abstract out... | 20:26 |
| SpamapS | nibalizer: would <%= hiera('ipmi_passwords')['server-host-name'] %> have a chance at working? | 20:26 |
| nibalizer | I mean I think you'd ened like scope.funcions.hiera() or something like that | 20:27 |
| nibalizer | there is some obtuse syntax for using puppet functions inside erb | 20:27 |
| clarkb | jeblair: ya I think thats the way we should proceed | 20:27 |
| nibalizer | but what causes you to not want the hiera call to be in the site.pp? | 20:27 |
| *** zul has joined #openstack-infra | 20:28 | |
| nibalizer | also if it's easier, mabye write code that doesn't work, and push it for review, and maybe then I can see the easy way to optimize it to not suck | 20:28 |
| SpamapS | nibalizer: http://paste.ubuntu.com/12120439/ | 20:28 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Commit to test stable/kilo multinode https://review.openstack.org/214334 | 20:28 |
| *** achanda has quit IRC | 20:28 | |
| SpamapS | nibalizer: this does not feel scalable. | 20:28 |
| *** achanda has joined #openstack-infra | 20:28 | |
| nibalizer | SpamapS: no, it doesn't | 20:29 |
| SpamapS | and thats with my silly 6 char hostnames | 20:29 |
| clarkb | so instead you should pass a hash | 20:29 |
| nibalizer | yes hash | 20:29 |
| clarkb | then hiera has a hash with all those values and you pass a single thing | 20:29 |
| nibalizer | ^^ | 20:29 |
| clarkb | then your erb says give me a hash that I will explode | 20:29 |
| nibalizer | (boom) | 20:29 |
| *** jeadr has quit IRC | 20:29 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Updated configuration for OIDC https://review.openstack.org/214335 | 20:29 |
| *** geoffarnold has quit IRC | 20:30 | |
| *** alivigni has quit IRC | 20:30 | |
| *** jeadr has joined #openstack-infra | 20:30 | |
| *** geoffarnold has joined #openstack-infra | 20:30 | |
| *** tjones has joined #openstack-infra | 20:30 | |
| SpamapS | kk that does sound better | 20:31 |
| *** nadya has quit IRC | 20:32 | |
| anteaya | SpamapS: are you at the ops meetup? | 20:32 |
| anteaya | I don't see you | 20:32 |
| SpamapS | anteaya: nope | 20:32 |
| SpamapS | anteaya: spent all last week in Taiwan.. not fond of 2 weeks of travel at a time. :) | 20:33 |
| SpamapS | well, spent Sun - Thu in Taiwan anyway | 20:33 |
| clarkb | did you land during the typhoon? | 20:33 |
| SpamapS | still experiencing occasional low grade jet lag narcolepsy | 20:33 |
| SpamapS | clarkb: 24h after | 20:33 |
| SpamapS | clarkb: delayed my flight by 5h | 20:33 |
| * Shrews slaps SpamapS awake | 20:33 | |
| clarkb | it was named after the group of people that conquered pohnpei | 20:34 |
| anteaya | Shrews: wow, yeah I can understand that | 20:34 |
| anteaya | SpamapS: sorry Shrews | 20:34 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Updated configuration for OIDC https://review.openstack.org/214335 | 20:34 |
| SpamapS | There were mailboxes tilted 45 degrees, and some smashed ancient buildings. Otherwise, that place is built like a fortress ... they don't seem to care much about 170mph wind | 20:34 |
| clarkb | watching CNN mispronounce it was pretty funny | 20:34 |
| fungi | clarkb: what was the name? | 20:35 |
| clarkb | Soudelor | 20:35 |
| fungi | and yeah, cnn and twc are terrible at pronunciating | 20:36 |
| clarkb | https://en.wikipedia.org/wiki/Saudeleur_Dynasty | 20:36 |
| jeblair | clarkb: cool. actioned! and we're getting names https://etherpad.openstack.org/p/PAO-ops-logging | 20:36 |
| clarkb | jeblair: woot | 20:36 |
| clarkb | fungi: they ruled from nan madol which is apparently inspiration for bits and pieces of cthulu things | 20:37 |
| fungi | clarkb: indeed it is | 20:38 |
| *** sputnik13 has quit IRC | 20:38 | |
| clarkb | also I highly recommend visiting if ancient ruins interest you | 20:38 |
| *** puranamr has joined #openstack-infra | 20:39 | |
| fungi | absolutely on my list | 20:39 |
| clarkb | fungi: the trick is you want to go at high tide and by floating vehicle of some sort if possible | 20:40 |
| *** flepied has joined #openstack-infra | 20:40 | |
| *** changbl has quit IRC | 20:40 | |
| clarkb | then you can wander all the canals, if you go by land you end up doing a lot of wading in the swamp which is not the most fun | 20:40 |
| *** angdraug has quit IRC | 20:41 | |
| fungi | clarkb: oh, good point. we have some places here which are similarly only conveniently navigable at high tide. to go on foot you wear rubber waders up to your chest | 20:41 |
| *** angdraug has joined #openstack-infra | 20:41 | |
| *** sharikapongubala has quit IRC | 20:42 | |
| * fungi needs to hurry up and get a canoe/sea kayak | 20:42 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add tools to make infra-cloud inventory templates https://review.openstack.org/214279 | 20:42 |
| SpamapS | nibalizer: ^ | 20:42 |
| SpamapS | nibalizer: note make-infracloud-hiera.py | 20:43 |
| *** sharikapongubala has joined #openstack-infra | 20:43 | |
| *** gyee has joined #openstack-infra | 20:43 | |
| SpamapS | if you can follow it.. it spits out an erb with <%= @ipmi_passwords[hostname] %> | 20:43 |
| SpamapS | not sure if that will work.. | 20:43 |
| SpamapS | probably have to quote the hostname actually | 20:43 |
| *** mmmpork has left #openstack-infra | 20:44 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add tools to make infra-cloud inventory templates https://review.openstack.org/214279 | 20:45 |
| sigmavirus24 | Have there been updates to the dsvm image lately? | 20:45 |
| greghaynes | sigmavirus24: they are made daily if it works | 20:46 |
| *** mpopow has joined #openstack-infra | 20:46 | |
| jeblair | clarkb, pabelanger, SpamapS: people just started talking about graphite and grafana and i said "hey, grafana, grafyaml, infra cloud....". there was positive response to creating a repo to share grafyaml configs for grafana for use in monitoring openstack deployments | 20:47 |
| *** esker has quit IRC | 20:47 | |
| *** mmmpork has joined #openstack-infra | 20:47 | |
| sigmavirus24 | hm, os-ansible-deployment is seeing weird cinder problems on hpcloud boxen in their gate and it only started recently. I'm about to start dissecting recent commits, but the weird thing is that it works on rackspace cloud boxen | 20:47 |
| SpamapS | jeblair: \o/ .. no deployment seems alive to me without graphs. | 20:48 |
| clarkb | sigmavirus24: they do have different flavors with different disk diszes | 20:48 |
| clarkb | sigmavirus24: its possible we stopped doing the ephemeral dirve formating on one for some reason | 20:48 |
| *** warpc__ has quit IRC | 20:48 | |
| sigmavirus24 | clarkb: oh, are the differences noted somewhere? | 20:48 |
| jeblair | clarkb, pabelanger, SpamapS: i also discovered grafyaml is hard to pronounce and i need to practice it at home when no one is listening. | 20:49 |
| clarkb | sigmavirus24: devstack-gate's setup function documents via code | 20:49 |
| openstackgerrit | Pradeep Kilambi proposed openstack/requirements: Add jsonpath-rw-ext to global requirements https://review.openstack.org/214331 | 20:49 |
| nibalizer | SpamapS: I'm pretty sure this isn't right | 20:49 |
| SpamapS | nibalizer: if I had ipmi_passwords => hiera('ipmi_passwords', 'XXX'), in my box manifest, that won't produce what I want? | 20:50 |
| pabelanger | jeblair: nice to hear | 20:50 |
| nibalizer | it depends on what you are trying to create | 20:50 |
| SpamapS | assuming my hiera bits have been successfully put in the right place for hiera to pick them up. | 20:50 |
| nibalizer | if the idea is this file will be then fed to puppet for erb template expansion, its pretty close | 20:50 |
| SpamapS | nibalizer: yes, that template will create the inventory file for bifrost | 20:51 |
| nibalizer | but then you shouldn't be calling it a hiera file at all | 20:51 |
| *** pabelanger has quit IRC | 20:51 | |
| nibalizer | you should be calling it bifrost_inventor.js.erb | 20:51 |
| SpamapS | that way we can have inventory in git w/o the passwords | 20:51 |
| nibalizer | ya | 20:51 |
| SpamapS | Can call it whatever | 20:51 |
| *** Sukhdev has quit IRC | 20:52 | |
| SpamapS | the program prints an erb on stdout, and reads/writes the hiera yaml at the path you give it. | 20:52 |
| nibalizer | oh im seeing that now | 20:52 |
| mriedem | dhellmann: do you have jurisdiction over creating branches for python-swiftclient? | 20:52 |
| *** derekh has quit IRC | 20:52 | |
| SpamapS | the idea being you take the fully assembled inventory, edit it with the tools we have for autodiscovery, run this, it makes a hiera for you and an erb template to store in git. | 20:52 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: Updated OpenStackId configuration https://review.openstack.org/214344 | 20:52 |
| nibalizer | ya works for me | 20:53 |
| *** _kiran_ has quit IRC | 20:53 | |
| SpamapS | the only bummer about this is that hashes don't get sorted right in yaml/json ..so diffing these erbs will never work out | 20:54 |
| nibalizer | well on '<%= @ipmi_passwords[{}] %>'.format(name) do you think we need to introduce quotes there? | 20:54 |
| nibalizer | oh wait maybe you fixed that already | 20:54 |
| SpamapS | nibalizer: I pushed a new version :) | 20:54 |
| nibalizer | ya you did | 20:54 |
| fungi | jeblair: you can pretty clearly see on the various cacti graphs for etherpad where the backup started around 05:30 and stopped when you killed it. apparently most of the cpu and all of the disk i/o was from that alone | 20:55 |
| SpamapS | oh wait | 20:55 |
| SpamapS | json.dumps has a sort_keys=True | 20:55 |
| SpamapS | huzzah | 20:55 |
| nibalizer | so what I reccomend we do is put a 'bifrost' entry in your hiera.yaml so that this file can be just bifrost.yaml | 20:55 |
| crinkle | nibalizer: ++ | 20:56 |
| nibalizer | and will be perpindicular to the common.yaml that will inevitably grow | 20:56 |
| nibalizer | and ofc there is public data in system-config now | 20:56 |
| nibalizer | if you want to try to use that | 20:56 |
| nibalizer | though I'm not sure what the puppetmaster configuration is on your controler node | 20:56 |
| SpamapS | nibalizer: I don't really know what that means. | 20:57 |
| jeblair | purp: what's the eplite bug you're looking at? | 20:57 |
| nibalizer | SpamapS: crinkle does so possibly you won't have to | 20:57 |
| purp | jeblair: https://github.com/ether/etherpad-lite/issues/1763 | 20:57 |
| jeblair | fungi: wow | 20:57 |
| crinkle | nibalizer: I was playing with https://review.openstack.org/#/c/209632/ to try to avoid puppetmaster config | 20:57 |
| nibalizer | but hiera.yaml is the configuration file for hiera, then it points to other yaml files | 20:57 |
| SpamapS | nibalizer: +1 | 20:57 |
| purp | jeblair: "Slow down with lots of different contributions" | 20:57 |
| SpamapS | nibalizer: OH THAT, yes, I did actually read that ;) | 20:57 |
| purp | This one seems related to the editor, but there are three others mentioned in there which have perf implications. | 20:58 |
| *** xyang1 has quit IRC | 20:58 | |
| nibalizer | crinkle: well I think this actually doesn't need that? local hiera with puppet apply on the infra-cloud-controller node is all we need | 20:58 |
| crinkle | nibalizer: right | 20:59 |
| purp | jeblair: One that sdague filed and was closed as dupe: https://github.com/ether/etherpad-lite/issues/1380 | 20:59 |
| *** julim has quit IRC | 21:00 | |
| *** zul has quit IRC | 21:00 | |
| *** sridhar_ram has quit IRC | 21:01 | |
| *** krtaylor has quit IRC | 21:01 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add tools to make infra-cloud inventory templates https://review.openstack.org/214279 | 21:02 |
| SpamapS | ok, so next hiera question.. | 21:02 |
| SpamapS | So I'm going to have an inventory per-site | 21:02 |
| *** jeadr has quit IRC | 21:02 | |
| purp | jeblair: am no longer convinced that's strongly related. | 21:02 |
| *** smarcet has quit IRC | 21:03 | |
| *** jeadr has joined #openstack-infra | 21:03 | |
| *** sridhar_ram has joined #openstack-infra | 21:04 | |
| nibalizer | SpamapS: i have a quesiton for you also | 21:04 |
| SpamapS | should I put _the whole thing_ in two hieras, one which is just the json stripped of passwords, and the other which is just the passwords, and then make an erb that merges those and spits out the inventory.json.. and then each site gets its own set of inventory hieras? | 21:04 |
| nibalizer | the infra-cloudcontrollers, how are they managed | 21:04 |
| hogepodge | etherpad :-( | 21:04 |
| nibalizer | are they puppet apply with local stuff? do they phone home to puppetmaster.openstack.org with puppet agent? | 21:05 |
| SpamapS | nibalizer: see the patch above 214279 in the dep chain. :) | 21:05 |
| *** stevemar has quit IRC | 21:05 | |
| SpamapS | nibalizer: they'll work like any other infra box | 21:05 |
| fungi | hogepodge: still? or again? or just you? | 21:05 |
| nibalizer | SpamapS: I kindof like how you are doing it now | 21:06 |
| SpamapS | nibalizer: yeah, just trying to figure out what to name the erb that I check in. | 21:06 |
| nibalizer | where there is a file very close to the input for bifrost, and puppet sneaks in at the last second and makes a little string replace | 21:06 |
| nibalizer | maybe put the cloudname in it? | 21:07 |
| *** aysyd has quit IRC | 21:07 | |
| nibalizer | what are we calling them HP1 and HP2? | 21:07 |
| SpamapS | I guess files/bifrost/inventory-hpuswest.json.erb .. and then have the name of the template to use as a parameter for the module? | 21:07 |
| SpamapS | the hiera can just be all the passwords since it is keyed by fqdn | 21:07 |
| nibalizer | ya then puppet will do content => template("bifrost/inventory/${cloudname}.json.erb") | 21:08 |
| SpamapS | we have to tell hiera to do deep merging on any hashes though, right? | 21:08 |
| *** puranamr has quit IRC | 21:09 | |
| nibalizer | SpamapS: https://review.openstack.org/#/c/208751/17/modules/openstack_project/manifests/infracloud/bifrost.pp if that takes a parameter we can expand the variable down on line 34 | 21:09 |
| *** amotoki has joined #openstack-infra | 21:10 | |
| SpamapS | nibalizer: sweet I like that. k | 21:10 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Try alternate usernames to get logs from overcloud https://review.openstack.org/214139 | 21:10 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Remove heat pin https://review.openstack.org/214348 | 21:10 |
| nibalizer | SpamapS: merging on hashes (in the hiera language) doesn't really apply here | 21:10 |
| nibalizer | we should just put in the hash | 21:10 |
| nibalizer | and not get fancy | 21:10 |
| *** thorst has quit IRC | 21:10 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Remove neutron pin https://review.openstack.org/214349 | 21:10 |
| hogepodge | fungi: capacity problem at the ops meetup | 21:11 |
| *** rfolco has quit IRC | 21:13 | |
| SpamapS | nibalizer: my thinking is just that in order to expand inventory I have to get access to the existing entries. But if I can just add another yaml with an ipmi_passwords hash, the deep merge will handle it. | 21:13 |
| SpamapS | nibalizer: agreed, over thinking. :) | 21:13 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Tell hiera to read bifrost.yaml https://review.openstack.org/214353 | 21:13 |
| *** krtaylor has joined #openstack-infra | 21:14 | |
| nibalizer | SpamapS: https://review.openstack.org/#/c/214353/ will enable hiera such that if /opt/system-config/hieradata/production/bifrost.yaml exists, hiera will try to load keys from it | 21:15 |
| *** amotoki has quit IRC | 21:15 | |
| crinkle | nibalizer: so regarding "how are they managed" and ^ I think https://review.openstack.org/#/c/209632/ is relevant and feedback would be nice | 21:15 |
| *** flashgordon is now known as jogo | 21:15 | |
| crinkle | I don't think "they'll work like any other infra box" fully applies here | 21:15 |
| *** stevemar has joined #openstack-infra | 21:16 | |
| crinkle | the module environment issue comes up | 21:16 |
| nibalizer | yes | 21:16 |
| *** mrmartin has quit IRC | 21:17 | |
| fungi | hogepodge: yeah, we've been monitoring that, but what specifically? several things have been done today to help try to improve the situation. does it seem to be helping? | 21:17 |
| nibalizer | well so there are, broadly, two classes of infra-cloud boxes right, one is the controllers, one is everything else | 21:17 |
| nibalizer | I'm wondering if we can't sneak the controllers into the main production environment | 21:17 |
| crinkle | I think there's three, one is a bifrost controller | 21:17 |
| hogepodge | fungi: it's not as terrible as it was this morning, but still some 503'ing | 21:17 |
| crinkle | nibalizer: we need a modern mysql module for the controllers | 21:18 |
| hogepodge | you can just see it as a general :-( at this point | 21:18 |
| *** ldnunes has quit IRC | 21:18 | |
| clarkb | crinkle: its probably relatively safe to bumo our mysql module since we trove most of the prod services | 21:18 |
| clarkb | if that helps | 21:18 |
| *** bharath_ has joined #openstack-infra | 21:18 | |
| crinkle | clarkb: good to know | 21:18 |
| *** puranamr has joined #openstack-infra | 21:19 | |
| crinkle | I still think it'll be generally less painful to separate them, which could be done in a number of ways | 21:19 |
| nibalizer | ok | 21:19 |
| fungi | hogepodge: after what looks like a lunchtime lull, the server stats we're monitoring for etherpad.openstack.org look a lot more healthy at least | 21:19 |
| *** bharath has quit IRC | 21:20 | |
| jeblair | fungi, hogepodge: yeah, my observation is that we're seeing relatively frequent disconnects, but generally working after reconnect, whereas this morning it was decidedly nonfunctional | 21:21 |
| jeblair | fungi, clarkb: can you hop on https://etherpad.openstack.org/p/PAO-ops-meetup ? | 21:22 |
| nibalizer | crinkle: this looks good | 21:22 |
| *** zul has joined #openstack-infra | 21:22 | |
| clarkb | jeblair: am there | 21:22 |
| clarkb | but splitting time with cross project meeting | 21:22 |
| jeblair | fungi, clarkb: would be interesting to note if you see similar disconnect rates | 21:22 |
| *** doug-fish has quit IRC | 21:22 | |
| crinkle | nibalizer: it is very rough and I hadn't considered until now 1) puppetdb and 2) hiera | 21:22 |
| *** doug-fish has joined #openstack-infra | 21:22 | |
| nibalizer | crinkle: yea puppetdb i have a plan for sortof | 21:22 |
| nibalizer | just need time to see if its viable | 21:22 |
| nibalizer | mordred: and I sketched out the plan for hiera essentiall | 21:23 |
| crinkle | nibalizer: awesomesauce | 21:23 |
| nibalizer | its not complicated, just need an ansible task to copy the right files to the target host before it runs puppet | 21:23 |
| fungi | jeblair: hogepodge: yeah, wondering if the disconnects now are local network unhappiness at the venue | 21:23 |
| hogepodge | fungi: maybe? | 21:23 |
| fungi | i'm monitoring my browser tab to that pad | 21:23 |
| crinkle | but yeah my hope with that patch is maybe there's a way to start out infracloud on masterless without waiting for the whole infra to move over | 21:23 |
| crinkle | but there's lots of other options too | 21:24 |
| fungi | hogepodge: jeblair: what's the disconnect frequency you're observing? | 21:24 |
| hogepodge | fungi: I'm limiting myself to one etherpad tab fwiw | 21:24 |
| purp | fungi hogepodge jeblair: I tethered this morning to test that theory, still had 503s. Haven't tried this afternoon. | 21:24 |
| *** jeadr has quit IRC | 21:24 | |
| hogepodge | fungi: it's intermittent. At least once every five minutes, but once I hit one reconnecting takes a long while | 21:24 |
| jeblair | purp: ah thanks. so much else was going on this morning (backups, memory leak, stray abiword) it's worth retesting | 21:24 |
| jeblair | fungi: been a while since i've seen one | 21:25 |
| purp | Agree. Am about to tether again and see if I get clear. | 21:25 |
| fungi | so far i've not seen any issues with that pad | 21:25 |
| *** tiswanso has quit IRC | 21:25 | |
| jeblair | i'm also not seeing mass enter/exits in the log | 21:25 |
| *** jeadr has joined #openstack-infra | 21:25 | |
| fungi | and see 56 editors on it | 21:25 |
| *** mestery has quit IRC | 21:25 | |
| nibalizer | crinkle: I think we can infracloud masterless | 21:25 |
| nibalizer | oaky reviewed the thing | 21:26 |
| *** erikwilson_ has quit IRC | 21:26 | |
| nibalizer | SpamapS: under crinkle's plan, you'll need a hiera.yaml (the hiera configuration file) and the infracloud bifrost.yaml on the bifrost node itself | 21:26 |
| nibalizer | not on the openstack-infra puppetmaster | 21:26 |
| SpamapS | nibalizer: thats how we'll operate until we are ready to tax infra-root and let workloads run on it. ;) | 21:27 |
| nibalizer | SpamapS: no I think thats actually just how it will run... | 21:27 |
| nibalizer | I mean the puppetmaster might start overwriting your file with the exact same file every 20 minutes | 21:27 |
| nibalizer | but that would be a transparent change | 21:27 |
| nibalizer | https://review.openstack.org/#/c/164870/ set it up so every node has a hiera.yaml but it hit merge conflict before it landed | 21:28 |
| nibalizer | okay really I'm leaving now | 21:28 |
| purp | Okay, tethered and re-opened a kerjillion tabs. Let's see if I get the 503 again. | 21:29 |
| clarkb | purp: also be sure you don't open the same pad twice | 21:29 |
| clarkb | purp: that will break and upstream didn't seem interested in fixing it | 21:29 |
| jeblair | nice | 21:29 |
| *** sputnik13 has joined #openstack-infra | 21:29 | |
| hogepodge | fungi: over the last 10 minutes? things are better | 21:30 |
| jeblair | it's cause i'm tailing logs | 21:30 |
| purp | jeblair: deduped. | 21:30 |
| purp | sorry, clarkb: deduped. | 21:30 |
| hogepodge | anecdotally | 21:30 |
| *** tonytan4ever has quit IRC | 21:30 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add bifrost inventory management procedures https://review.openstack.org/214279 | 21:30 |
| SpamapS | crinkle: ^^ | 21:31 |
| SpamapS | greghaynes: ^ | 21:31 |
| SpamapS | Hopefully that makes sense, and you all can chip away the unnecessary pieces, and we can have a well defined procedure for adding nodes. | 21:31 |
| crinkle | SpamapS: neat | 21:32 |
| crinkle | SpamapS: you can also just take over 208751 if you want | 21:32 |
| purp | clarkb: first fact: eight pads couldn't reconnect. Closed 7. Reconnected to main pad. | 21:32 |
| *** otter768 has joined #openstack-infra | 21:32 | |
| purp | Just reopened 10. | 21:33 |
| *** sputnik13 has quit IRC | 21:33 | |
| purp | all connected and seem to be working. Let's see what happens now. | 21:34 |
| clarkb | the one jeblair aksed me to open is still happy | 21:34 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add config flags for data source configuration https://review.openstack.org/213811 | 21:34 |
| purp | clarkb: Yeah, I opened ten of the links there, too. Recreating the problem from this morning, if I can. | 21:34 |
| greghaynes | SpamapS: anything about how to obtain the raw-inventory.json? | 21:34 |
| greghaynes | also ty for documenting | 21:35 |
| jeblair | we are split now, but soon we will recombine and likely have a higher single-pad load | 21:35 |
| *** cbader has quit IRC | 21:35 | |
| *** stevemar has quit IRC | 21:36 | |
| fungi | yeah, still no disconnects to the PAO-ops-meetup pad for me | 21:36 |
| purp | clarkb fungi: anything interesting in the server log? Just curious. | 21:37 |
| *** otter768 has quit IRC | 21:37 | |
| *** geoffarnold has quit IRC | 21:37 | |
| *** erikwilson_ has joined #openstack-infra | 21:38 | |
| *** erikwilson_ has quit IRC | 21:38 | |
| jeblair | i'm tailing it and have not seen mass enter/exists. that's all i'm watching for right now though. | 21:38 |
| jeblair | (that was easy to spot this morning) | 21:38 |
| purp | Ahhh. Lost connection to logging page ... | 21:38 |
| fungi | yeah, nothing exciting there yet | 21:38 |
| fungi | still no disconnects for me | 21:38 |
| clarkb | jeblair: yup and it should pause too | 21:39 |
| purp | Still tethered. Network is slow (crappy reception) but able to load rich pages from common servers. | 21:39 |
| clarkb | at least it did that earlier the log would stop scrolling for a minute then you would get the LEAVE ENTERs | 21:39 |
| purp | About 5 pages lost connection and reconnected. | 21:40 |
| clarkb | I opened 8 and all 8 opened | 21:40 |
| clarkb | the last 2 were slower | 21:40 |
| purp | May have been a spike in my tether's latency. | 21:40 |
| purp | Lost connection again. | 21:41 |
| purp | While it was reconnecting, I saw "Waiting for available socket" | 21:42 |
| clarkb | hrm its possible that we are hitting the apache worker/thread limit | 21:42 |
| *** tonytan4ever has joined #openstack-infra | 21:42 | |
| *** sileht has quit IRC | 21:43 | |
| clarkb | but based on cacti I don't think so | 21:43 |
| clarkb | it should be set to 2048 iirc and we are doing half that according to cacti | 21:43 |
| clarkb | maybe its a fd ulimit? | 21:43 |
| *** olaph has joined #openstack-infra | 21:43 | |
| *** dkranz has quit IRC | 21:44 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools: Bump pbr version to match g-r https://review.openstack.org/214363 | 21:44 |
| anteaya | jbryce: can you come to the large deployments room? | 21:44 |
| *** amitgandhinz has quit IRC | 21:44 | |
| anteaya | jbryce: there is a discussion about openstack's legal obligations | 21:45 |
| fungi | clarkb: well, apache is proxying these, so needs 2 fd per connection | 21:45 |
| fungi | one in, one out | 21:45 |
| purp | jeblair: You're still connected, can edit a page? | 21:45 |
| anteaya | jbryce: would be nice for you to at least be aware/weigh in | 21:45 |
| clarkb | fungi: right but the grand ottal is ~1k according to cacti | 21:45 |
| *** shashank_hegde has quit IRC | 21:45 | |
| fungi | ahh, got it | 21:45 |
| clarkb | fungi: of all connections | 21:45 |
| jeblair | purp: yep, just edited pao-ops-meetup | 21:45 |
| jeblair | clarkb: that's a suspicous number | 21:46 |
| *** bpokorny has quit IRC | 21:46 | |
| fungi | i still haven't lost connectivity to PAO-ops-meetup | 21:46 |
| *** bpokorny_ has joined #openstack-infra | 21:46 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=116&rra_id=all | 21:46 |
| clarkb | that graph fwiw | 21:46 |
| jeblair | hrm, server-status doesn't work on that host :/ | 21:46 |
| nigelb | fungi: our attack was in the same time region. | 21:46 |
| purp | I've lost connection off and on, but more likely sparse bandwidth on tether. I lose all at once, regain all at once. | 21:46 |
| purp | No 503s. | 21:46 |
| nigelb | And they didn't stop after I did the ban on mailman lists. It's just we became unaffected by it. | 21:47 |
| jeblair | clarkb: hrm, looks like we regularly go much higher | 21:47 |
| SpamapS | crinkle: I may do that. :) | 21:48 |
| *** bpokorny_ has quit IRC | 21:48 | |
| *** mriedem has quit IRC | 21:48 | |
| jeblair | purp: i think hotel connectivity may be more reliable at the moment, i haven't seen a problem in a while | 21:48 |
| *** hdd has quit IRC | 21:48 | |
| fungi | nigelb: right, i think they were likely hitting every public mailman instance on the 'net | 21:48 |
| SpamapS | greghaynes: No I don't actually know how you got raw-inventory.json. ;) | 21:48 |
| *** bpokorny has joined #openstack-infra | 21:49 | |
| fungi | nigelb: and as i said earlier, the + isn't really exploiting anything about mailman, it's just a convenient means of providing for a near-infinite number of e-mail addresses that all go to the same inbox, for senders whose mta implements such a feature | 21:49 |
| purp | jeblair: flipped back. | 21:49 |
| *** zul has quit IRC | 21:49 | |
| *** tiswanso has joined #openstack-infra | 21:50 | |
| nigelb | fungi: Yep! Someone really wanted to fill up that one dude's inbox. | 21:50 |
| purp | Managed to get an error (o5SFgkkpeV8hi9mTVX1v) but no 503. | 21:50 |
| fungi | nigelb: i have a strong feeling that the e-mail address which was being subscribed over and over was the actual victim of a denial-of-service attack, and our mailman servers were simply being used as convenient reflectors/amplifiers | 21:50 |
| nigelb | Exactly. | 21:50 |
| *** sthillma has joined #openstack-infra | 21:50 | |
| fungi | all subscription requests were coming to us via http post, not inbound smtp... i assume was the same for you as well | 21:50 |
| nigelb | yep | 21:51 |
| nigelb | And really distributed. | 21:51 |
| nigelb | I think the largest requests from one ip was 50 | 21:51 |
| fungi | nigelb: yeah, we saw slightly over 5000 ip addresses. i assume those were all compromised systems in a moderate-sized bot army | 21:52 |
| greghaynes | SpamapS: heh, ok, I can document that as a parent patch? | 21:52 |
| *** bpokorny has quit IRC | 21:52 | |
| fungi | nigelb: all the ip addresses i checked were in assignments belonging to china mobile, though i have no reason to believe it was really only originating from that one service provider | 21:52 |
| *** bpokorny has joined #openstack-infra | 21:52 | |
| clarkb | fungi: I need sanity check, can you go to nodepool and grep fo rthe single held 2-node instance then try sshing to it? | 21:52 |
| *** asettle has joined #openstack-infra | 21:53 | |
| clarkb | fungi: ssh asks for my password which makes me thing my rekeying exploded or something odd is going on | 21:53 |
| nigelb | fungi: I had multiple providers across US and Europe as well. | 21:53 |
| *** puranamr has quit IRC | 21:53 | |
| anteaya | regXboi: can you show jeblair the graphs you made for neutron? | 21:53 |
| nigelb | I checked a bunch of them for abuse contacts and eventually gave up. | 21:53 |
| fungi | nigelb: good to know. i only spot-checked a handful | 21:53 |
| clarkb | pleia2: ^ maybe you too? | 21:53 |
| anteaya | regXboi: I got telling jeblair about it and got pulled away, so he has a bit of context | 21:53 |
| fungi | clarkb: i am able to log into 104.239.173.107 devstack-trusty-2-node-rax-iad-4377574 | 21:53 |
| SpamapS | greghaynes: I think it would make sense as an update to that patch. Feel free to ammend it. Just let me know when you do. | 21:54 |
| fungi | clarkb: seems fine | 21:54 |
| clarkb | fungi: can you check what my authorized keys looks like? | 21:54 |
| greghaynes | SpamapS: ok | 21:54 |
| pleia2 | clarkb: checking | 21:54 |
| nigelb | I'm *slightly* worried that there's really not much we can do in terms of preventing this again. Other than watch out for the attack. | 21:54 |
| clarkb | hrm maybe I only had the old key added to this agent | 21:54 |
| clarkb | and I derped in the opposite way to what I thought I had derped | 21:54 |
| regXboi | anteaya: is jeblair at PAO or in channel? | 21:54 |
| fungi | nigelb: ends in hycFHXlDrlBgfFJDe5At clark@work | 21:54 |
| fungi | er, clarkb ^ | 21:54 |
| clarkb | fungi: thanks | 21:54 |
| jeblair | regXboi: both :) | 21:54 |
| fungi | sorry nigelb | 21:54 |
| nigelb | heh | 21:54 |
| pleia2 | clarkb: yeah, is ok for me too | 21:54 |
| regXboi | jeblair: which breakout room are you in? | 21:54 |
| * regXboi is in Med II in the back corner | 21:55 | |
| jeblair | regXboi: logging. maybe let's meet at coffee break. i'll be wearing a hat. | 21:55 |
| regXboi | ok, I've got reading glasses, a tan heather shirt and my handle on my badge :) | 21:55 |
| jeblair | with 'jeblair' on the hat. | 21:55 |
| regXboi | :) | 21:55 |
| nigelb | fungi: But seriously, I'm glad I'm not alone. I spent all of morning yesterday figuring out what went wrong with the server. | 21:55 |
| regXboi | I can see about finding you | 21:55 |
| clarkb | fungi: pleia2 gah that was it I thought I had the new key loaded but had the old one loaded | 21:55 |
| fungi | nigelb: anyway, it looked to me like >99.9% of the browser strings claimed to be firefox 39 on windows 7, so i expect that was fallacious, which implies some software making the post calls from those systems rather than something like a cross-site request forgery | 21:56 |
| nigelb | The only external indication I had were postfix queue alerts and load average alerts. | 21:56 |
| clarkb | so I had inverted my expected state and ya. anyways all better now thank you for looking | 21:56 |
| nigelb | Yeah, I noticed that too. That was the common factor. | 21:56 |
| anteaya | regXboi: jeblair is at the operators meetup | 21:56 |
| anteaya | regXboi: he is wearing a 3 piece suit, fedora and bow tie | 21:56 |
| nigelb | The good news is that a subscription ban in mailman for the correct regex is effective in making this attack not affect systems. | 21:57 |
| regXboi | anteaya: I konw about the fedora :) | 21:57 |
| fungi | nigelb: interestingly, the only indication we had was moderators of subscriber-confirmed lists complaining that they had hundreds of subscription requests to approve from nearly-identical e-mail addresses | 21:57 |
| purp | regXboi: He's in Salon A. | 21:57 |
| *** dannywil_ has joined #openstack-infra | 21:57 | |
| anteaya | regXboi: :) | 21:57 |
| fungi | nigelb: by the time it came to my attention, it had already subsided for 4-5 hours | 21:57 |
| anteaya | purp: thanks | 21:57 |
| purp | <--- right there (if you're sitting in my chair, which you're not) | 21:57 |
| nigelb | I caught it before it subsided because timezones :D | 21:57 |
| nigelb | so I was deleting the queue | 21:58 |
| * jeblair can verify that purp is in his chair | 21:58 | |
| hogepodge | jeblair: fungi way more stable now | 21:58 |
| nigelb | and having it refilled every minute. | 21:58 |
| *** dannywil_ has quit IRC | 21:58 | |
| *** dannywil_ has joined #openstack-infra | 21:58 | |
| fungi | hogepodge: yep, still no disconnects for me. haven't seen even one so far | 21:58 |
| regXboi | purp: danke | 21:59 |
| nigelb | I'm going to setup some alerts to catch it if it happens again on our systems. | 21:59 |
| nigelb | It's kinda nice to hold hands across organizations/projects for something like this :) | 21:59 |
| fungi | nigelb: absolutely! if you see something we're happy to check our systems for confirmation so we can tell you it's really not just you ;) | 22:00 |
| pleia2 | nigelb: ^5 | 22:00 |
| *** dannywilson has quit IRC | 22:01 | |
| nigelb | fungi / pleia2 ^5! yay! | 22:01 |
| *** dprince has quit IRC | 22:02 | |
| *** dannywil_ has quit IRC | 22:02 | |
| *** tiswanso has quit IRC | 22:03 | |
| *** btully has quit IRC | 22:04 | |
| *** minwang2 has quit IRC | 22:04 | |
| *** mpopow has quit IRC | 22:04 | |
| *** tiswanso has joined #openstack-infra | 22:05 | |
| *** minwang2 has joined #openstack-infra | 22:05 | |
| *** doug-fish has quit IRC | 22:05 | |
| *** shashank_hegde has joined #openstack-infra | 22:05 | |
| *** gyee has quit IRC | 22:06 | |
| *** ybathia has quit IRC | 22:07 | |
| *** sputnik13 has joined #openstack-infra | 22:07 | |
| *** btully has joined #openstack-infra | 22:08 | |
| dougwig | clarkb: final revert merged, the nodepool vpn issue should magically disappear. | 22:08 |
| clarkb | dougwig: thats I will recheck to confirm | 22:08 |
| clarkb | s/thats/thanks | 22:08 |
| *** tiswanso has quit IRC | 22:10 | |
| *** amotoki has joined #openstack-infra | 22:10 | |
| fungi | etherpad is still rock-solid for me | 22:11 |
| fungi | but now i must eat | 22:11 |
| clarkb | same here | 22:11 |
| dougwig | fungi: and by rock, he means pick up a rock, look at it, and that's your etherpad. | 22:11 |
| pleia2 | hehe | 22:11 |
| asselin_ | clarkb, mordred Shrews akraminsky so when I have the json to use projectId instead of tenantID, I run into other issues... http://paste.openstack.org/show/421057/ | 22:12 |
| clarkb | hpcloud pypi mirror is done syncing | 22:12 |
| clarkb | lifeless: ^ all should be well once again | 22:12 |
| *** sabeen has joined #openstack-infra | 22:12 | |
| lifeless | clarkb: thanks | 22:13 |
| *** regXboi has quit IRC | 22:13 | |
| *** abregman has quit IRC | 22:13 | |
| *** sabeen2 has joined #openstack-infra | 22:14 | |
| *** angdraug has quit IRC | 22:14 | |
| *** amotoki has quit IRC | 22:15 | |
| *** austin81 has left #openstack-infra | 22:16 | |
| *** sabeen has quit IRC | 22:17 | |
| *** sputnik13 has quit IRC | 22:17 | |
| *** glauco_ has quit IRC | 22:17 | |
| *** annegentle has quit IRC | 22:17 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:18 | |
| *** camunoz has joined #openstack-infra | 22:20 | |
| *** bswartz has joined #openstack-infra | 22:22 | |
| *** pradk has quit IRC | 22:24 | |
| *** saurabhs has joined #openstack-infra | 22:24 | |
| *** pcrews has joined #openstack-infra | 22:25 | |
| clarkb | dougwig: https://jenkins04.openstack.org/job/gate-dsvm-nodepool-nv/10/console should get through devstacking | 22:26 |
| *** sputnik13 has joined #openstack-infra | 22:26 | |
| openstackgerrit | Somay Jain proposed openstack-infra/jenkins-job-builder: Support for CVS SCM using CVS plugin https://review.openstack.org/203685 | 22:26 |
| *** sputnik13 has quit IRC | 22:29 | |
| *** ivar-lazzaro has quit IRC | 22:30 | |
| *** jeadr has quit IRC | 22:30 | |
| *** shashank_hegde has quit IRC | 22:31 | |
| *** angela-s has joined #openstack-infra | 22:31 | |
| *** jeadr has joined #openstack-infra | 22:31 | |
| *** mpopow has joined #openstack-infra | 22:31 | |
| *** regXboi has joined #openstack-infra | 22:33 | |
| *** shashank_hegde has joined #openstack-infra | 22:33 | |
| *** regXboi has joined #openstack-infra | 22:33 | |
| *** geoffarnold has joined #openstack-infra | 22:35 | |
| *** puranamr has joined #openstack-infra | 22:35 | |
| *** gordc has quit IRC | 22:36 | |
| *** sputnik13 has joined #openstack-infra | 22:38 | |
| *** tjones has quit IRC | 22:40 | |
| *** darrenc is now known as darrenc_afk | 22:40 | |
| *** tjones has joined #openstack-infra | 22:40 | |
| mordred | asselin_: that is very strange | 22:42 |
| clarkb | heh 10-15 minute sfor devstack is a lie | 22:42 |
| asselin_ | mordred, but I still don't know why that json change is needed...looks it's been that way for a while.. | 22:43 |
| *** ashleighfarnham has quit IRC | 22:44 | |
| SpamapS | hm.. so now that I have a way to make an inventory.. I need to make some boxes and test it out | 22:45 |
| mordred | asselin_: I don't either - also, is admin the project id and not the project name? (just brainstorming) | 22:45 |
| mtreinish | clarkb: yep it is, we've constantly had to bump that number in d-g | 22:45 |
| asselin_ | both | 22:45 |
| asselin_ | sorry, project name and user name | 22:45 |
| mtreinish | clarkb: it's sort of tracked with: http://status.openstack.org//elastic-recheck/gate.html#1311066 | 22:46 |
| SpamapS | greghaynes: which box has all the boxes you deployed in ironic? | 22:47 |
| greghaynes | SpamapS: I unenrolled all but one in the bastion1 so I could cycle fast on some glean stuff a while back | 22:47 |
| *** cbader has joined #openstack-infra | 22:47 | |
| *** eharney has quit IRC | 22:47 | |
| greghaynes | but its super easy to just reenroll al the things | 22:48 |
| *** baoli has joined #openstack-infra | 22:49 | |
| greghaynes | that is the host I was using though, the .3 address | 22:50 |
| *** tlian2 has joined #openstack-infra | 22:50 | |
| *** romcheg has quit IRC | 22:50 | |
| *** romcheg has joined #openstack-infra | 22:51 | |
| *** tlian has quit IRC | 22:51 | |
| *** romcheg has quit IRC | 22:51 | |
| *** cbader has quit IRC | 22:53 | |
| mordred | asselin_: no, I mean, you're calling it project_id but it's "admin" | 22:53 |
| *** baoli has quit IRC | 22:53 | |
| mordred | asselin_: was checking to make sure it was id and not name - I don't think it matters - just thinking out loud | 22:53 |
| mordred | asselin_: can you paste your clouds.yaml (password removed of course) | 22:54 |
| asselin_ | mordred, ok I see what you mean, I can try projectName. (that was just a guess at josn) | 22:54 |
| *** tonytan4ever has quit IRC | 22:55 | |
| *** asettle is now known as asettle-afk | 22:56 | |
| *** darrenc_afk is now known as darrenc | 22:56 | |
| *** angela-s has quit IRC | 22:57 | |
| clarkb | mtreinish: looks like 20 minutes is closer to reality | 22:57 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Switch Horizon to Javascript-jobs https://review.openstack.org/214373 | 22:57 |
| SpamapS | greghaynes: kk thanks | 22:57 |
| asselin_ | mordred, well turns out that's just an ignored field....I see now where it gets the service catalog, from the response, in this case it's empty | 22:58 |
| asselin_ | mordred, I don't use cloudconfig currently, it's in the nodepool.yaml file | 22:58 |
| mordred | asselin_: ah. ok. I wonder if we broke somethign with that | 22:58 |
| mordred | asselin_: thanks - that's helpful to know where to look | 22:58 |
| asselin_ | mordred, cool, thanks for your help | 22:59 |
| regXboi | anteaya: ping ... https://wiki.openstack.org/wiki/QA/CodeSprintLibertyFortCollins#Registration ... smile | 22:59 |
| mordred | asselin_: and I forget what you said in scrollback. are you now running latest nodepool and latest shade? | 23:00 |
| adam_g | anyone know an easy + automated way to seed a fresh gerrit with an admin user? | 23:00 |
| asselin_ | mordred, yes | 23:00 |
| clarkb | adam_g: you login with your mouse | 23:00 |
| adam_g | clarkb, read: automated | 23:01 |
| adam_g | :) | 23:01 |
| clarkb | adam_g: there has been talk upstream of fixing that but the only other alternative is to mess with the db ahead of time and hope that they don't break you via a migration or other fun | 23:01 |
| clarkb | adam_g: so do that at your own risk | 23:01 |
| greghaynes | adam_g: javascript? :p | 23:01 |
| mordred | asselin_: k. cool | 23:01 |
| adam_g | clarkb, yeah, i had to do something like that in the past. it was a PITA | 23:01 |
| asselin_ | mordred, ok I see it now! it should be tenantName | 23:01 |
| clarkb | adam_g: also be very careful using gerrit >=2.9 | 23:02 |
| clarkb | adam_g: jgit is terribly broken and can eat your repos | 23:02 |
| jeblair | fyi, the current session is: https://etherpad.openstack.org/p/PAO-ops-containers-for-deployment | 23:02 |
| *** cbader has joined #openstack-infra | 23:02 | |
| jeblair | i mention that not only because we might get a lot of ppl on the etherpad | 23:02 |
| *** pcrews has quit IRC | 23:02 | |
| jeblair | but also the content may be of interest here :) | 23:02 |
| clarkb | adam_g: if you do find something better than messing with the db and risking a migration break definitely let us know | 23:03 |
| greghaynes | clarkb: couldnt you just web scrape essentially | 23:03 |
| asselin_ | mordred, akraminsky curl output with json updated to tenantName http://paste.openstack.org/show/421132/ | 23:03 |
| clarkb | maybe but the gwt elements change too | 23:03 |
| adam_g | clarkb, oh, thanks for the tip | 23:03 |
| greghaynes | clarkb: ayre, right | 23:03 |
| fungi | jeblair: thanks. i still haven't disconnected from that one fwiw | 23:04 |
| mordred | asselin_: I see publicURL for glance in there | 23:04 |
| mordred | oh - you know what ... | 23:05 |
| *** camunoz has quit IRC | 23:05 | |
| asselin_ | mordred, here's my debug output | 23:05 |
| asselin_ | http://paste.openstack.org/show/421133/ | 23:05 |
| *** camunoz has joined #openstack-infra | 23:05 | |
| mordred | service_name=self.cloud_config.get_service_name( | 23:06 |
| mordred | service_key), | 23:06 |
| asselin_ | tenant id needs to be none for tenant name to get used | 23:06 |
| *** wenlock has quit IRC | 23:06 | |
| fungi | adam_g: i believe i confirmed not long ago that you can use the baked in "Gerrit Code Review" username (you use the gerrit ssh api host key as the user authentication key) to call the user create command | 23:06 |
| *** tiswanso has joined #openstack-infra | 23:06 | |
| fungi | adam_g: in essence, there _is_ a default admin user in gerrit with ssh-only authentication access | 23:07 |
| adam_g | fungi, oh nice | 23:07 |
| *** geoffarnold has quit IRC | 23:07 | |
| jeblair | fungi: neato... and if you do that, it triggers the first-user-is-admin feature? (and the next user does not get that?) | 23:08 |
| fungi | adam_g: ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Gerrit Code Review'@127.0.0.1 gerrit create-account whatever | 23:08 |
| fungi | jeblair: that part i do not know | 23:08 |
| fungi | but should be easy to test | 23:08 |
| jeblair | ya | 23:08 |
| *** shashank_hegde has quit IRC | 23:08 | |
| *** annegentle has joined #openstack-infra | 23:08 | |
| adam_g | fungi, sweet! ill give that a shot next time around, thanks | 23:08 |
| fungi | i merely confirmed on review-dev a while back that you can actually use that meta-account to create accounts | 23:08 |
| fungi | the api commands allowed to it are limited, so i wasn't certain without testing | 23:09 |
| clarkb | ffungi iirc you cant | 23:09 |
| clarkb | you can once you setup the server but not for first user | 23:09 |
| *** annegentle has quit IRC | 23:09 | |
| fungi | oh, weird | 23:09 |
| clarkb | this is the thing upstrean intends to fix | 23:09 |
| clarkb | but until jgit works... | 23:10 |
| *** annegentle has joined #openstack-infra | 23:10 | |
| fungi | yeah, what i haven't tested is trying that on a fresh deployment with no first user created through the webui | 23:10 |
| *** shashank_hegde has joined #openstack-infra | 23:10 | |
| clarkb | zaro ^ | 23:10 |
| *** dannywilson has joined #openstack-infra | 23:10 | |
| *** smccully has quit IRC | 23:11 | |
| *** amotoki has joined #openstack-infra | 23:11 | |
| *** dimtruck is now known as zz_dimtruck | 23:13 | |
| asselin_ | mordred, akraminsky so when I hack into keystone client I can get nodepool to work again: http://paste.openstack.org/show/421134/ | 23:13 |
| asselin_ | now to find the correct fix | 23:13 |
| *** zz_dimtruck is now known as dimtruck | 23:13 | |
| clarkb | adam_g https://groups.google.com/forum/m/#!searchin/repo-discuss/Admin/repo-discuss/4OaOoEVOy8A | 23:13 |
| mordred | asselin_: why not just pass it as tenant-name instead of tenant-id? | 23:14 |
| clarkb | looks like clas may have bitten agaib | 23:15 |
| *** gyee has joined #openstack-infra | 23:15 | |
| clarkb | but that plugin may work | 23:15 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: JavaScript in Infra https://review.openstack.org/198884 | 23:15 |
| akraminsky | mordred, tenant_name doesn't exist | 23:15 |
| *** rm_work is now known as rm_work|away | 23:15 | |
| akraminsky | or is None | 23:15 |
| mordred | but asselin_ is hacking keystoneclient to pass tenant_name and that made it work | 23:16 |
| asselin_ | mordred, I need to see where that happens. | 23:16 |
| *** puranamr has quit IRC | 23:16 | |
| *** amotoki has quit IRC | 23:16 | |
| asselin_ | I need to trace the code now to see why tenantid is not set to None | 23:16 |
| mordred | well, it's likely in os-client-config | 23:17 |
| asselin_ | mordred, ok will look there | 23:17 |
| asselin_ | mordred, to answer your previous question, here's the sanitized part of nodepool.yaml we use http://paste.openstack.org/show/421135/ | 23:17 |
| mordred | I believe we collapse tenant_id and tenant_name because it was our experience that the clouds did not care and we could use them interchangable | 23:17 |
| *** glauco_ has joined #openstack-infra | 23:17 | |
| asselin_ | mordred, apperently not... | 23:18 |
| *** tiswanso has quit IRC | 23:18 | |
| mordred | asselin_: yah. funny that | 23:18 |
| mordred | asselin_: so- nodepool is going to pass it in as project-id always | 23:18 |
| mordred | which shade will either pass in as project-id or tenant-id depending on whether or not you're using v2 or v3 | 23:19 |
| asselin_ | mordred, really?? | 23:19 |
| adam_g | clarkb, interesting. the db route may just be the easiest for me, i only plan on doing this as part of initial boostrap. and since you've scared me into to pegging to a lower gerrit version im not worried about migrations | 23:19 |
| asselin_ | ok I see | 23:19 |
| asselin_ | so I guess the other option is to update my project-id from 'admin' to the numeric id | 23:20 |
| asselin_ | but that's not convenient | 23:20 |
| mordred | asselin_: that's probably a short-term solution | 23:20 |
| *** dims_ has joined #openstack-infra | 23:20 | |
| mordred | asselin_: I think we should stop treating project-id and project-name as interchangable | 23:20 |
| asselin_ | why doesn't shade prefer project-name over project-id | 23:21 |
| mordred | it's os-client-config that does the args processing | 23:21 |
| mordred | but I believe it's because we'd found that passing everything in as project_id would make keystoneclient do the right thing | 23:21 |
| *** achanda has quit IRC | 23:22 | |
| mordred | so - project_name should be the winner from occ | 23:22 |
| mordred | looking through the code | 23:22 |
| *** dims has quit IRC | 23:22 | |
| akraminsky | lol fix_backwards_madness | 23:23 |
| *** dimtruck is now known as zz_dimtruck | 23:23 | |
| mordred | which means that even you setting project-id in nodepool, you should be winding up with tenant_name being passed to keystoneclient | 23:23 |
| *** geoffarnold has joined #openstack-infra | 23:24 | |
| *** glauco_ has quit IRC | 23:24 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added ACL for ironic-webclient https://review.openstack.org/214381 | 23:24 |
| asselin_ | so much magic | 23:25 |
| mordred | yes | 23:25 |
| mordred | it's very sadening | 23:25 |
| *** asettle-afk has quit IRC | 23:26 | |
| mordred | akraminsky, asselin_ https://review.openstack.org/214382 | 23:27 |
| *** dannywilson has quit IRC | 23:28 | |
| *** cbader has quit IRC | 23:28 | |
| asselin_ | mordred, thanks, will test it now | 23:28 |
| mordred | asselin_: if we land that ^^ and then we move your config to using clouds.yaml instead of embedded auth things | 23:28 |
| mordred | asselin_: then you should be able to be more explicit about what you're configuring | 23:29 |
| *** chlong has quit IRC | 23:29 | |
| mordred | and we can remove one piece of magic | 23:29 |
| mordred | from the pile | 23:29 |
| krotscheck | Is this how new ACL lists are added? https://review.openstack.org/#/c/214381/ | 23:29 |
| mordred | asselin_: I say you need to move to clouds.yaml - because nodepool does not grok anything other than the project-id parameter | 23:29 |
| *** achanda has joined #openstack-infra | 23:30 | |
| *** achanda has quit IRC | 23:30 | |
| asselin_ | mordred, I'm still curious to know what change broke our setup | 23:30 |
| mordred | me too | 23:31 |
| mordred | I don't see anything in your setup that should have broken | 23:31 |
| *** achanda has joined #openstack-infra | 23:31 | |
| asselin_ | mordred, yes, was hoing to move to clouds.yaml when migrating to the common-ci nodepool solution | 23:31 |
| *** changbl has joined #openstack-infra | 23:31 | |
| mordred | unless keystoneclient made a release and it stopped doing the thing that was strangely working before? | 23:31 |
| mordred | nod | 23:31 |
| asselin_ | I'm going to debug a bit more | 23:32 |
| mordred | kk | 23:32 |
| mordred | sorry I'm not more direct debugging help - I wish I had a good local reproduction | 23:32 |
| *** otter768 has joined #openstack-infra | 23:33 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin https://review.openstack.org/204615 | 23:35 |
| *** puranamr has joined #openstack-infra | 23:35 | |
| anteaya | krotscheck: yes | 23:36 |
| anteaya | krotscheck: do you plan to be creating branches in ironic-webclient? if no, please remove line 3 | 23:36 |
| mordred | reed: where did your shade version of the first-app doc get published? | 23:36 |
| *** jeadr has quit IRC | 23:37 | |
| mordred | reed: the only one i can find is the libcloud one | 23:37 |
| fungi | clarkb: that url to the repo-discuss group seems to not have any content | 23:37 |
| fungi | clarkb: though i'm not immediately grokking how a cla could impede the "gerrit code review" built-in account | 23:37 |
| clarkb | soneone wrote code now cant push upstream | 23:38 |
| *** ZZelle-off has quit IRC | 23:38 | |
| *** otter768 has quit IRC | 23:38 | |
| fungi | clarkb: oh, unrelated to the initial user creation problem? | 23:39 |
| reed | mordred, no, because (I learned today) that to be published one needs to complete the full tutorial, including the scaling and other documents | 23:39 |
| reed | mordred, so far I only have finished the gettingstarted.rst file | 23:39 |
| mordred | reed: oh. oy | 23:39 |
| adam_g | clarkb, are you all still doing similar db stuff to sync lp group members to gerrit? | 23:39 |
| mordred | adam_g: nope | 23:40 |
| reed | mordred, fifieldt today suggested to enable a draft/ site but that requires more playing with publishing tools... | 23:40 |
| adam_g | mordred, howzit work now | 23:40 |
| mordred | adam_g: gerrit groups are managed in gerrit - lp groups only have meaning in launchpad | 23:40 |
| mordred | reed: nod | 23:40 |
| adam_g | mordred, i see | 23:41 |
| clarkb | fungi well its code to fix initial user problem | 23:41 |
| fungi | clarkb: aha, i get it. i thought you were implying that the initial user problem was cla-related | 23:41 |
| fungi | rather i guess someone with a fix to the problem is refusing to agree to google's cla or something (wonder why i can't see any content at that url, maybe google only shows you groups threads if you have an account now?) | 23:42 |
| adam_g | fungi, https://groups.google.com/forum/#!topic/repo-discuss/4OaOoEVOy8A | 23:42 |
| *** jeadr has joined #openstack-infra | 23:43 | |
| fungi | adam_g: thanks--that one's working | 23:44 |
| *** jeadr has quit IRC | 23:46 | |
| fungi | adam_g: clarkb: i don't see any indication in that thread that the "gerrit code review" account won't work for that | 23:46 |
| fungi | discussion there seems to revolve around the chicken-and-egg problem of not having an account with which to create an account | 23:47 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add new contrail-neutron-plugin project https://review.openstack.org/214389 | 23:47 |
| *** rm_work|away is now known as rm_work | 23:47 | |
| fungi | except there is an existing pseudo-account, it's just so poorly documented i doubt many of the gerrit devs even know it exists | 23:47 |
| fungi | anyway, worth a try | 23:48 |
| clarkb | I think its impkied | 23:48 |
| clarkb | we need this other thing because existibg does not work | 23:48 |
| *** jeadr has joined #openstack-infra | 23:48 | |
| *** dangers is now known as dangers_away | 23:48 | |
| *** geoffarnold has quit IRC | 23:49 | |
| adam_g | fungi, fwiw just tried that, apparently i need a passphrase to use that host key | 23:49 |
| *** geoffarnold has joined #openstack-infra | 23:49 | |
| fungi | adam_g: i hope not. otherwise gerrit won't be able to load that key at startup either | 23:50 |
| *** shashank_hegde has quit IRC | 23:51 | |
| jeblair | ooh, people have heard of infra cloud! | 23:51 |
| Sam-I-Am | is that like a cloud of infra? | 23:52 |
| fungi | it's like a nebulous thing full of nebulous stuff | 23:52 |
| jeblair | cloud! | 23:52 |
| Sam-I-Am | can it run in a container? | 23:52 |
| fungi | a container can run in it | 23:52 |
| jeblair | you can't contain a cloud | 23:53 |
| fungi | life finds a way | 23:53 |
| openstackgerrit | Merged openstack-infra/system-config: Use new api for jenkins masters credentials https://review.openstack.org/199737 | 23:53 |
| *** thorst has joined #openstack-infra | 23:54 | |
| greghaynes | fungi: someone sent me this about dib the other day http://i.imgur.com/hAYd2tI.gif | 23:54 |
| *** sergmelikyan has quit IRC | 23:55 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm publish to eslint-config-openstack https://review.openstack.org/199725 | 23:56 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Switched npm-run-test to npm-test https://review.openstack.org/212154 | 23:56 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm publish job. https://review.openstack.org/199715 | 23:56 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm tarball upload job https://review.openstack.org/212153 | 23:56 |
| *** SumitNaiksatam has quit IRC | 23:56 | |
| fungi | greghaynes: so, so true | 23:56 |
| *** sergmelikyan has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
