| *** e0ne has quit IRC | 00:00 | |
| *** etoews has joined #openstack-infra | 00:07 | |
| *** adalbas has quit IRC | 00:07 | |
| *** dkehn is now known as dkehn_afk | 00:08 | |
| *** morganfainberg_Z is now known as morganfainberg | 00:09 | |
| *** etoews has quit IRC | 00:12 | |
| *** rfolco has joined #openstack-infra | 00:13 | |
| *** rfolco has quit IRC | 00:14 | |
| *** rfolco has joined #openstack-infra | 00:15 | |
| *** matsuhashi has joined #openstack-infra | 00:20 | |
| *** etoews has joined #openstack-infra | 00:24 | |
| *** fifieldt has joined #openstack-infra | 00:24 | |
| *** jhesketh has quit IRC | 00:33 | |
| *** talluri has joined #openstack-infra | 00:41 | |
| *** morganfainberg is now known as morganfainberg_Z | 00:41 | |
| *** aconrad has joined #openstack-infra | 00:42 | |
| *** talluri has quit IRC | 00:45 | |
| *** mattoliverau has quit IRC | 00:50 | |
| *** yamahata has joined #openstack-infra | 00:52 | |
| *** mattoliverau has joined #openstack-infra | 00:53 | |
| *** etoews has quit IRC | 00:57 | |
| *** e0ne has joined #openstack-infra | 00:58 | |
| *** unicell has joined #openstack-infra | 00:59 | |
| *** e0ne has quit IRC | 01:03 | |
| stevebaker | Hey, I wonder if https://review.openstack.org/90628 could be fast-tracked into tempest to see if it helps with https://bugs.launchpad.net/heat/+bug/1297560 | 01:16 |
|---|---|---|
| uvirtbot | Launchpad bug 1297560 in tempest "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,New] | 01:16 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Bump check-rally-dsvm-{name} job timeout https://review.openstack.org/90630 | 01:19 |
| *** yamahata has quit IRC | 01:20 | |
| *** yamahata has joined #openstack-infra | 01:21 | |
| *** gondoi is now known as zz_gondoi | 01:23 | |
| *** zz_gondoi is now known as gondoi | 01:24 | |
| *** etoews has joined #openstack-infra | 01:26 | |
| *** saschpe has quit IRC | 01:32 | |
| *** saschpe has joined #openstack-infra | 01:32 | |
| *** thomasem_ has joined #openstack-infra | 01:33 | |
| *** thomasem has quit IRC | 01:35 | |
| *** dkranz has quit IRC | 01:39 | |
| *** yaguang has joined #openstack-infra | 01:44 | |
| msabramo | With pbr, is it possible to express that a package needs unittest2 for Python ≤ 2.6 but nor for Python > 2.6? | 01:47 |
| msabramo | normally I would do that with python code in setup.py but with setup.cfg, it's declarative so that doesn't seem possible | 01:47 |
| *** nosnos has joined #openstack-infra | 01:50 | |
| *** rfolco has left #openstack-infra | 01:55 | |
| *** dims has quit IRC | 01:56 | |
| *** rfolco_ has joined #openstack-infra | 01:57 | |
| *** e0ne has joined #openstack-infra | 01:58 | |
| *** e0ne has quit IRC | 02:00 | |
| mattoliverau | msabramo: Not that I know of, but you force it to check the version of python, i.e requires-python >=2.6 | 02:02 |
| mattoliverau | But I don't know how'd you do the split otherwise. | 02:03 |
| *** alexpilotti has quit IRC | 02:08 | |
| *** dims has joined #openstack-infra | 02:09 | |
| mattoliverau | msabramo: On closer look, you might be able to do: Requires-Dist: unittest2; python_version == '2.4' or python_version == '2.5' | 02:10 |
| mattoliverau | msabramo: based off PEP 345 (http://is.gd/CXwPTJ) | 02:10 |
| *** msabramo has quit IRC | 02:12 | |
| *** etoews has quit IRC | 02:23 | |
| *** e0ne has joined #openstack-infra | 02:25 | |
| *** e0ne has quit IRC | 02:26 | |
| *** thomasem_ has quit IRC | 02:26 | |
| *** msabramo has joined #openstack-infra | 02:27 | |
| msabramo | thanks, mattoliverau - I'll take a look at that PEP! | 02:38 |
| msabramo | it doesn't look like pbr supports that from the source code | 02:40 |
| *** gokrokve has quit IRC | 02:41 | |
| msabramo | but it looks like it might support having requirements-py2.txt and requirements-py3.txt | 02:41 |
| *** zhiyan_ is now known as zhiyan | 02:47 | |
| *** msabramo has quit IRC | 02:56 | |
| *** e0ne has joined #openstack-infra | 02:58 | |
| *** e0ne has quit IRC | 03:02 | |
| *** msabramo has joined #openstack-infra | 03:09 | |
| *** asettle is now known as alex-lunch | 03:11 | |
| *** gokrokve has joined #openstack-infra | 03:11 | |
| *** gokrokve_ has joined #openstack-infra | 03:14 | |
| *** gokrokve has quit IRC | 03:16 | |
| *** dims has quit IRC | 03:17 | |
| *** dims has joined #openstack-infra | 03:18 | |
| anteaya | mattoliverau: hey there | 03:18 |
| anteaya | mattoliverau: if anyone asks we are upgrading gerrit at 1600 utc: https://wiki.openstack.org/wiki/GerritUpgrade | 03:19 |
| *** gokrokve_ has quit IRC | 03:19 | |
| anteaya | mattoliverau: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033070.html | 03:20 |
| anteaya | and welcome back from your vacation | 03:20 |
| anteaya | I am about to hit the hay | 03:20 |
| anteaya | ttyl | 03:20 |
| mattoliverau | anteaya: hey! thanks, OK will let people know :) | 03:23 |
| StevenK | Oh, that's today? | 03:39 |
| StevenK | Well, tomorrow at 2am for me, so eh. :-) | 03:40 |
| *** gondoi is now known as zz_gondoi | 03:41 | |
| *** afazekas has quit IRC | 03:47 | |
| *** matsuhashi has quit IRC | 03:54 | |
| *** e0ne has joined #openstack-infra | 03:58 | |
| *** e0ne has quit IRC | 04:02 | |
| *** etoews has joined #openstack-infra | 04:05 | |
| *** alex-lunch has quit IRC | 04:07 | |
| *** asettle has joined #openstack-infra | 04:08 | |
| *** etoews has quit IRC | 04:09 | |
| *** nosnos has quit IRC | 04:10 | |
| *** asettle has quit IRC | 04:12 | |
| *** alexandra has joined #openstack-infra | 04:12 | |
| *** alexandra is now known as asettle | 04:13 | |
| *** gokrokve has joined #openstack-infra | 04:14 | |
| *** gokrokve has quit IRC | 04:19 | |
| *** sdake_ has joined #openstack-infra | 04:21 | |
| *** sdake_ has quit IRC | 04:21 | |
| *** sdake_ has joined #openstack-infra | 04:21 | |
| *** asalkeld has joined #openstack-infra | 04:24 | |
| asalkeld | hi I added stackforge/solum-dashboard recently | 04:25 |
| asalkeld | it's here: http://git.openstack.org/cgit/stackforge/solum-dashboard/ | 04:25 |
| asalkeld | but not on github | 04:25 |
| asalkeld | https://github.com/stackforge/solum-dashboard <- 404 | 04:25 |
| asalkeld | any thing I can check? | 04:26 |
| mordred | asalkeld: probably an error in the manage-projects script - I can take a look at it tomorrow | 04:26 |
| asalkeld | cool, thanks mordred | 04:26 |
| *** msabramo has quit IRC | 04:32 | |
| *** msabramo has joined #openstack-infra | 04:32 | |
| *** asalkeld has left #openstack-infra | 04:40 | |
| *** starmer_ has joined #openstack-infra | 04:43 | |
| *** etoews has joined #openstack-infra | 04:43 | |
| *** etoews has quit IRC | 04:48 | |
| *** afazekas has joined #openstack-infra | 04:49 | |
| *** UtahDave has joined #openstack-infra | 04:52 | |
| *** e0ne has joined #openstack-infra | 04:58 | |
| *** e0ne has quit IRC | 05:01 | |
| *** matsuhashi has joined #openstack-infra | 05:04 | |
| *** chandan_kumar has joined #openstack-infra | 05:06 | |
| *** yfried_ has joined #openstack-infra | 05:07 | |
| *** gokrokve has joined #openstack-infra | 05:07 | |
| *** gokrokve has quit IRC | 05:10 | |
| *** matsuhashi has quit IRC | 05:10 | |
| *** matsuhashi has joined #openstack-infra | 05:11 | |
| *** nosnos has joined #openstack-infra | 05:13 | |
| *** matsuhashi has quit IRC | 05:17 | |
| *** matsuhashi has joined #openstack-infra | 05:17 | |
| *** amotoki has joined #openstack-infra | 05:20 | |
| *** zhiyan is now known as zhiyan_ | 05:26 | |
| *** Longgeek has joined #openstack-infra | 05:31 | |
| *** chandan_kumar has quit IRC | 05:31 | |
| *** zhiyan_ is now known as zhiyan | 05:33 | |
| *** praneshp has quit IRC | 05:41 | |
| *** gokrokve has joined #openstack-infra | 05:41 | |
| *** rcarrill` has joined #openstack-infra | 05:41 | |
| *** chandan_kumar has joined #openstack-infra | 05:42 | |
| *** rcarrillocruz has quit IRC | 05:43 | |
| *** gokrokve has quit IRC | 05:46 | |
| *** lazy_prince has joined #openstack-infra | 05:51 | |
| yfried_ | clarkb: ping | 05:53 |
| *** Longgeek has quit IRC | 05:56 | |
| *** e0ne has joined #openstack-infra | 05:58 | |
| openstackgerrit | Marc Abramowitz proposed a change to stackforge/python-jenkins: 100% test coverage https://review.openstack.org/90500 | 05:58 |
| *** e0ne has quit IRC | 06:00 | |
| *** Longgeek has joined #openstack-infra | 06:02 | |
| *** aconrad has quit IRC | 06:02 | |
| *** Longgeek has quit IRC | 06:02 | |
| *** Longgeek has joined #openstack-infra | 06:02 | |
| *** ildikov has joined #openstack-infra | 06:12 | |
| *** gokrokve has joined #openstack-infra | 06:14 | |
| yfried_ | Can someone point me in the right directrion? I'm looking for the log-pusher script described here http://ci.openstack.org/logstash.html and I can't find it. can some | 06:16 |
| *** gokrokve has quit IRC | 06:19 | |
| *** achuprin has joined #openstack-infra | 06:22 | |
| achuprin | Hi team! | 06:22 |
| *** e0ne has joined #openstack-infra | 06:25 | |
| *** rcarrillocruz has joined #openstack-infra | 06:25 | |
| *** rcarrill` has quit IRC | 06:26 | |
| *** jamielennox is now known as jamielennox|away | 06:31 | |
| *** jlibosva has joined #openstack-infra | 06:40 | |
| *** flaper87|afk is now known as flaper87 | 06:42 | |
| *** _nadya_ has joined #openstack-infra | 06:44 | |
| *** chandan_kumar has quit IRC | 06:46 | |
| *** skolekonov has joined #openstack-infra | 06:51 | |
| *** UtahDave has quit IRC | 06:51 | |
| *** _nadya_ has quit IRC | 06:53 | |
| *** jcoufal has joined #openstack-infra | 06:54 | |
| *** chandan_kumar has joined #openstack-infra | 06:54 | |
| *** maxbit has joined #openstack-infra | 06:57 | |
| *** _nadya_ has joined #openstack-infra | 06:59 | |
| *** matsuhashi has quit IRC | 07:01 | |
| *** matsuhashi has joined #openstack-infra | 07:03 | |
| *** ociuhandu has joined #openstack-infra | 07:03 | |
| *** e0ne has quit IRC | 07:05 | |
| *** hdd has joined #openstack-infra | 07:10 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Remove unused openstack common modules https://review.openstack.org/90658 | 07:12 |
| *** gokrokve has joined #openstack-infra | 07:14 | |
| mattoliverau | Hey achuprin, sorry for late response :) | 07:17 |
| mattoliverau | yfried_: The scripts don't seem to be in the repository anymore, I guess the documentation needs to be updated. I'll have a poke around and see what I can find. | 07:18 |
| *** ihrachyshka has joined #openstack-infra | 07:18 | |
| *** gokrokve has quit IRC | 07:18 | |
| *** ihrachyshka has quit IRC | 07:23 | |
| *** matsuhashi has quit IRC | 07:23 | |
| *** ihrachyshka has joined #openstack-infra | 07:23 | |
| *** matsuhashi has joined #openstack-infra | 07:25 | |
| yfried_ | mattoliverau: hi, thank you. I'm trying to set up logstash on my private ci, and I'm looking for any help there might be | 07:25 |
| yfried_ | mattoliverau: do you know of any other documentation? I have all of my logs are collected into a tar file at the end of a job. I'd like to extract this tar and ship everything to logstash | 07:27 |
| yfried_ | seems better that creating a logstash agent on every OS node in the deployement and another one on the jenkins, etc... | 07:28 |
| *** ihrachyshka_ has joined #openstack-infra | 07:29 | |
| *** _nadya_ has quit IRC | 07:32 | |
| BobBall | sdague: See https://review.openstack.org/#/c/90660/ when you're around - urgent fix to your change to get XenServer CI back running. Waiting on tests results first, of course... ;) | 07:32 |
| *** ihrachyshka has quit IRC | 07:32 | |
| *** shardy_afk is now known as shardy | 07:38 | |
| mattoliverau | yfried_: The scripts still exist, it looks like they have moved to a different module in the puppet configuration (config repo). You can find them here: http://is.gd/3VNUE3 | 07:46 |
| *** jooools has joined #openstack-infra | 07:46 | |
| mattoliverau | yfried_: they use the same file name as mentioned in the documentation | 07:46 |
| mattoliverau | yfried_: I'm not up to speed with the logstash stuff in infra. So sorry if I can't help too much. I'll start taking a look into though so I can better help people in the future :) | 07:47 |
| mattoliverau | Well I'm off for the night, have a great night/day y'all. | 07:53 |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** e0ne has joined #openstack-infra | 07:54 | |
| *** hashar has joined #openstack-infra | 07:54 | |
| *** jlibosva has quit IRC | 07:57 | |
| *** jlibosva has joined #openstack-infra | 07:59 | |
| yfried_ | mattoliverau: do you know of anyone who is up to speed with logstash? | 07:59 |
| *** ildikov_ has joined #openstack-infra | 07:59 | |
| *** ildikov has quit IRC | 08:02 | |
| *** _nadya_ has joined #openstack-infra | 08:02 | |
| *** derekh has joined #openstack-infra | 08:03 | |
| *** yassine has joined #openstack-infra | 08:09 | |
| *** zhiyan is now known as zhiyan_ | 08:11 | |
| *** gokrokve has joined #openstack-infra | 08:14 | |
| *** zhiyan_ is now known as zhiyan | 08:16 | |
| *** e0ne_ has joined #openstack-infra | 08:16 | |
| *** e0ne__ has joined #openstack-infra | 08:17 | |
| *** ildikov has joined #openstack-infra | 08:18 | |
| *** gokrokve has quit IRC | 08:19 | |
| *** e0ne has quit IRC | 08:19 | |
| *** reed has joined #openstack-infra | 08:20 | |
| *** e0ne_ has quit IRC | 08:20 | |
| *** pblaho has joined #openstack-infra | 08:21 | |
| *** jgallard has joined #openstack-infra | 08:21 | |
| *** ildikov_ has quit IRC | 08:21 | |
| *** jlibosva has quit IRC | 08:23 | |
| *** vponomaryov has joined #openstack-infra | 08:24 | |
| *** jlibosva has joined #openstack-infra | 08:26 | |
| *** Ryan_Lane has quit IRC | 08:32 | |
| *** starmer_ has quit IRC | 08:33 | |
| *** morganfainberg_Z is now known as morganfainberg | 08:34 | |
| *** darraghb has joined #openstack-infra | 08:38 | |
| *** chandan_kumar has quit IRC | 08:46 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added test for migrations https://review.openstack.org/90047 | 08:49 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Fixed xml output generation for python 2.7.5 https://review.openstack.org/90533 | 08:56 |
| *** dizquierdo has joined #openstack-infra | 08:56 | |
| *** ihrachyshka_ has quit IRC | 08:57 | |
| *** lazy_prince has left #openstack-infra | 08:59 | |
| *** chandan_kumar has joined #openstack-infra | 09:05 | |
| *** jp_at_hp has joined #openstack-infra | 09:05 | |
| *** etoews has joined #openstack-infra | 09:07 | |
| *** gokrokve has joined #openstack-infra | 09:14 | |
| *** etoews has quit IRC | 09:15 | |
| *** gokrokve has quit IRC | 09:19 | |
| *** morganfainberg is now known as morganfainberg_Z | 09:30 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added test for migrations https://review.openstack.org/90047 | 09:34 |
| *** mkerrin has quit IRC | 09:45 | |
| *** maxbit has quit IRC | 09:47 | |
| *** maxbit has joined #openstack-infra | 09:48 | |
| *** _nadya_ has quit IRC | 09:50 | |
| *** ociuhandu has quit IRC | 09:53 | |
| *** _nadya_ has joined #openstack-infra | 09:54 | |
| *** _nadya_ has quit IRC | 09:57 | |
| *** alexpilotti has joined #openstack-infra | 09:58 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Add some more Trove classifiers https://review.openstack.org/90674 | 09:59 |
| *** ihrachyshka has joined #openstack-infra | 10:00 | |
| *** matsuhashi has quit IRC | 10:05 | |
| *** nosnos has quit IRC | 10:05 | |
| *** _nadya_ has joined #openstack-infra | 10:06 | |
| sdague | BobBall: you shouldn't need L352 | 10:12 |
| *** ihrachyshka has quit IRC | 10:14 | |
| *** gokrokve has joined #openstack-infra | 10:14 | |
| *** jgallard has quit IRC | 10:14 | |
| BobBall | sdague: because PRECACHE_IMAGES is only set for the official gate? | 10:15 |
| sdague | it's actually not set anywhere | 10:16 |
| BobBall | oh :) | 10:16 |
| sdague | except in the list_images.sh tool | 10:16 |
| BobBall | OK - I'll remove 352. | 10:16 |
| sdague | it seems like we maybe want a 2nd case block around heat image | 10:17 |
| sdague | because without a working heat image, you can't run a bunch of the heat tests | 10:17 |
| BobBall | hmmm - I wonder why it worked before. Maybe all of the heat tests were exlcuded from the XS CI | 10:18 |
| *** gokrokve has quit IRC | 10:18 | |
| *** tchaypo has quit IRC | 10:20 | |
| BobBall | Maybe we should just disable heat for xenserver for now? | 10:21 |
| *** rcarrillocruz has quit IRC | 10:23 | |
| *** rcarrillocruz has joined #openstack-infra | 10:28 | |
| BobBall | sdague: I'm not sure I understand... was HEAT_FETCHED_TEST_IMAGE set somewhere else before? Because if not, then surely there was no heat image in devstack before your change? | 10:28 |
| sdague | it was | 10:29 |
| BobBall | ah | 10:29 |
| sdague | the conditional was a little different | 10:29 |
| sdague | my guess is the xen ci wasn't setting it at all | 10:30 |
| BobBall | ah - I see it in d-g | 10:30 |
| BobBall | Let me check | 10:30 |
| BobBall | seems that it was only set if DEVSTACK_GATE_TEMPEST_HEAT_SLOW was enabled | 10:31 |
| BobBall | which it wasn't | 10:31 |
| sdague | ok | 10:31 |
| sdague | so we should put another hypervisor case block around this | 10:31 |
| BobBall | or shift it into the hypervisor block above? | 10:31 |
| *** chandan_kumar has quit IRC | 10:32 | |
| BobBall | I think (?) that libvirt is the only hypervisor that supports qcow2 | 10:32 |
| BobBall | so that image can only be valid for libvirt since glance will reject it for everything else if properly configured? | 10:32 |
| *** tchaypo has joined #openstack-infra | 10:37 | |
| BobBall | sdague: OK - I propose to block out the enabling of HEAT so that it is only enabled for libvirt, and move the qcow2 image for heat into the libvirt section of the existing VIRT_DRIVER case statement | 10:38 |
| BobBall | does that make sense? | 10:38 |
| sdague | sure | 10:40 |
| *** tchaypo has quit IRC | 10:41 | |
| *** zhiyan is now known as zhiyan_ | 10:41 | |
| *** tchaypo has joined #openstack-infra | 10:43 | |
| BobBall | sdague: https://review.openstack.org/#/c/90660/6/stackrc make sense then? | 10:43 |
| *** ihrachyshka has joined #openstack-infra | 10:46 | |
| *** andreaf has joined #openstack-infra | 10:47 | |
| lifeless | BobBall: baremetal supports qcow2 | 10:48 |
| lifeless | BobBall: moderately sure lxc does as well | 10:48 |
| BobBall | drat. but thanks lifeless | 10:48 |
| lifeless | BobBall: ironic does too (buts its not merged yet) | 10:48 |
| BobBall | yeah - lxc is libvirt though isn't it? | 10:48 |
| sdague | BobBall: ok, so that change can't go in. Because we don't want to not turn on heat | 10:49 |
| BobBall | OK - I'll take the wimps way out and just disable heat for xenserver to avoid this | 10:49 |
| lifeless | not sure, gotta crash :). Another way to addrewss the issue woude to fix qcow support for other drivers. | 10:49 |
| BobBall | oh - sorry - I thought you said that heat couldn't work without that image? | 10:49 |
| lifeless | night! | 10:49 |
| sdague | BobBall: some of the heat tests don't work | 10:49 |
| BobBall | love to lifeless - but that's far too big a fix - we need the CI back running soon :) | 10:49 |
| sdague | so let's just do another case block | 10:49 |
| *** chandan_kumar has joined #openstack-infra | 10:49 | |
| sdague | if [[ "$ENABLED_SERVICES" =~ 'h-api' ]]; | 10:50 |
| sdague | case (hypervisor) | 10:50 |
| sdague | and add the images there | 10:50 |
| BobBall | and set the image for all non-XS hypervisors? or just libvirt + baremetal? | 10:50 |
| BobBall | well - maybe libvirt + * with xenserver setting to nothing | 10:50 |
| sdague | right now libvirt + barebetal is probably fine | 10:50 |
| sdague | HEAT_CFN_IMAGE_URL being set triggers a tempest variable later | 10:51 |
| sdague | which turns on a set of tests which need a HEAT_CFN image | 10:51 |
| BobBall | sdague: Maybe https://review.openstack.org/#/c/90660/ then | 10:55 |
| sdague | BobBall: yep, that's good | 10:56 |
| BobBall | OK - let's hope all the tests pass then | 10:56 |
| BobBall | thanks for the help | 10:56 |
| *** mburned_out is now known as mburned | 10:57 | |
| * BobBall sits and twiddles his thumbs until they pass | 10:57 | |
| *** ildikov_ has joined #openstack-infra | 11:01 | |
| *** mkerrin has joined #openstack-infra | 11:01 | |
| *** yaguang has quit IRC | 11:02 | |
| *** e0ne has joined #openstack-infra | 11:02 | |
| *** ildikov has quit IRC | 11:04 | |
| *** yamahata has quit IRC | 11:04 | |
| *** e0ne__ has quit IRC | 11:05 | |
| *** fifieldt has quit IRC | 11:07 | |
| *** fifieldt has joined #openstack-infra | 11:08 | |
| *** dizquierdo has quit IRC | 11:10 | |
| *** gokrokve has joined #openstack-infra | 11:14 | |
| *** gokrokve has quit IRC | 11:18 | |
| *** jaypipes has joined #openstack-infra | 11:19 | |
| *** viktors has joined #openstack-infra | 11:32 | |
| *** rcarrill` has joined #openstack-infra | 11:33 | |
| *** maxbit has quit IRC | 11:33 | |
| *** rcarrillocruz has quit IRC | 11:34 | |
| viktors | Hi folks! Please, could anyone review patch https://review.openstack.org/#/c/78955/ (Import oslo.db repository) ? Thanks! | 11:35 |
| *** dizquierdo has joined #openstack-infra | 11:39 | |
| *** akscram has quit IRC | 11:43 | |
| *** akscram has joined #openstack-infra | 11:44 | |
| *** maxbit has joined #openstack-infra | 11:44 | |
| *** e0ne has quit IRC | 11:45 | |
| *** e0ne has joined #openstack-infra | 11:45 | |
| *** tristanC has joined #openstack-infra | 11:48 | |
| *** mbacchi has joined #openstack-infra | 11:48 | |
| *** jgallard has joined #openstack-infra | 11:50 | |
| *** e0ne has quit IRC | 11:50 | |
| *** zz_gondoi is now known as gondoi | 11:50 | |
| *** maxbit has quit IRC | 11:55 | |
| *** bcrochet has quit IRC | 11:56 | |
| *** bcrochet has joined #openstack-infra | 11:56 | |
| *** ArxCruz has joined #openstack-infra | 11:57 | |
| *** bcrochet has quit IRC | 12:00 | |
| sdague | fungi back today? | 12:01 |
| sdague | nodepool looks odd | 12:01 |
| *** bcrochet has joined #openstack-infra | 12:02 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: add support for python builder https://review.openstack.org/90464 | 12:02 |
| *** Longgeek has quit IRC | 12:03 | |
| *** chandan_kumar has quit IRC | 12:06 | |
| *** rlandy has joined #openstack-infra | 12:08 | |
| *** gondoi is now known as zz_gondoi | 12:09 | |
| *** adalbas has joined #openstack-infra | 12:10 | |
| *** e0ne has joined #openstack-infra | 12:11 | |
| openstackgerrit | Thomas Leaman proposed a change to openstack/requirements: Move to newer version of python-swiftclient https://review.openstack.org/76170 | 12:11 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities https://review.openstack.org/85565 | 12:11 |
| *** jaypipes has quit IRC | 12:12 | |
| *** yamahata has joined #openstack-infra | 12:12 | |
| *** pdmars has joined #openstack-infra | 12:14 | |
| *** gokrokve has joined #openstack-infra | 12:14 | |
| *** pdmars_ has joined #openstack-infra | 12:16 | |
| *** Longgeek has joined #openstack-infra | 12:16 | |
| *** pdmars_ has quit IRC | 12:16 | |
| *** gokrokve has quit IRC | 12:18 | |
| *** pdmars has quit IRC | 12:19 | |
| *** pdmars has joined #openstack-infra | 12:20 | |
| *** aysyd has joined #openstack-infra | 12:21 | |
| *** maxbit has joined #openstack-infra | 12:22 | |
| hashar | ! [remote rejected] HEAD -> refs/publish/master (you are not allowed to upload merges) | 12:22 |
| hashar | seriously gerrit | 12:22 |
| hashar | :-( | 12:22 |
| *** chandan_kumar has joined #openstack-infra | 12:24 | |
| *** yamahata has quit IRC | 12:24 | |
| *** jaypipes has joined #openstack-infra | 12:26 | |
| *** yamahata has joined #openstack-infra | 12:27 | |
| *** miqui has joined #openstack-infra | 12:28 | |
| *** markmc has joined #openstack-infra | 12:31 | |
| *** yfried_ has quit IRC | 12:32 | |
| *** yfried__ has joined #openstack-infra | 12:32 | |
| *** lazy_prince2 has joined #openstack-infra | 12:36 | |
| *** lazy_prince2 has left #openstack-infra | 12:37 | |
| *** maxbit has quit IRC | 12:37 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 12:40 |
| fungi | sdague: back, yes. trying to catch up | 12:40 |
| fungi | hashar: are you trying to merge a feature branch in a project in gerrit? | 12:41 |
| sdague | fungi: welcome back! | 12:41 |
| hashar | fungi: I wanted to merge two changes with master (an octopus merge) | 12:41 |
| hashar | then put a patch on top of it :] | 12:41 |
| hashar | something like: git merge --no-ff pending_change_1 pending_change_2 && git cherry-pick my_new_feature | 12:41 |
| hashar | but I can wait :] | 12:42 |
| *** smarcet has joined #openstack-infra | 12:42 | |
| fungi | hashar: normally pushing merge commits into gerrit is not what you want. it's designed to sequentially merge rebased/cherry-picked/squashed commits since anything else is very hard to review | 12:42 |
| boris-42 | fungi hi | 12:42 |
| fungi | hi boris-42 | 12:42 |
| boris-42 | fungi how are you?) | 12:42 |
| hashar | fungi: yup that is painful. The merge was merely to let me work on a future state of the repo to avoid painful rebase later on. | 12:42 |
| hashar | fungi: I am sure I will survive it :] | 12:42 |
| fungi | boris-42: not sure yet. trying to absorb two weeks of scrollback and ml archives, plus it's the openstack project vacation week so technically infra's not here (except that we're upgrading gerrit in a few hours) | 12:43 |
| *** dims has quit IRC | 12:43 | |
| boris-42 | fungi prapering for upadte?) | 12:43 |
| sdague | fungi: I don't think most of the openstack project is on vacation :) | 12:43 |
| fungi | boris-42: yes | 12:43 |
| sdague | I think ttx needs to not suggest that next time | 12:43 |
| fungi | sdague: heh, clearly they didn't listen ;) | 12:44 |
| boris-42 | sdague fungi ahaha=) | 12:44 |
| sdague | fungi: well, once you get settled in, nodepool looks quite odd | 12:44 |
| sdague | in terms of nodes running vs. what it believes should be running | 12:44 |
| boris-42 | 117 lol | 12:44 |
| fungi | sdague: yeah, looking at nodepool i'm betting it's concept of what nodes are available doesn't match reality. probably rackspace rebooted it for us again | 12:44 |
| fungi | bad guess... up 5 days | 12:45 |
| boris-42 | fungi btw downtime should be just a couple of hrs? | 12:45 |
| *** timrc is now known as timrc-afk | 12:46 | |
| boris-42 | fungi btw could you take a look at this patch (already 8x+1 and 1x+2) https://review.openstack.org/#/c/89801/ and only 4 LOCs | 12:46 |
| fungi | boris-42: probably at least a couple hours. we've done multiple dry-runs of the upgrade process with review-dev.openstack.org but chances are some features may not be working completely when we first get it running (we're skipping ~4 release versions, from 2.4.x to 2.8.x) | 12:47 |
| boris-42 | fungi uhhh=) | 12:47 |
| boris-42 | fungi dangerous=) | 12:47 |
| fungi | also, if we end up having to roll back and restore the database from backups, it may be longer downtime still | 12:48 |
| *** etoews has joined #openstack-infra | 12:48 | |
| *** mfer has joined #openstack-infra | 12:48 | |
| fungi | anyway, i'm working on diagnosing nodepool's apparent schizophrenia for the moment so we can get jobs running again | 12:49 |
| *** eharney has joined #openstack-infra | 12:49 | |
| boris-42 | fungi ok | 12:50 |
| *** weshay has joined #openstack-infra | 12:51 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 12:51 |
| boris-42 | fungi btw successfully catch of issue with eventlet under load in keystone in gates http://logs.openstack.org/48/90248/8/check/check-rally-dsvm-rally/23c0d03/rally-plot/results.html.gz ! | 12:51 |
| *** ociuhandu has joined #openstack-infra | 12:52 | |
| *** dprince has joined #openstack-infra | 12:54 | |
| *** timrc-afk is now known as timrc | 12:55 | |
| *** zhiyan_ is now known as zhiyan | 12:56 | |
| *** dims has joined #openstack-infra | 12:56 | |
| *** rfolco_ has quit IRC | 12:56 | |
| fungi | sdague: still looking, but initial analysis suggests that nodepool's numbers are accurate, however it's definitely maxxed out on quotas and looks like it's adding/deleting a lot of nodes... there could perhaps be build failures at play | 12:56 |
| *** rfolco_ has joined #openstack-infra | 12:58 | |
| *** smarcet has quit IRC | 12:58 | |
| *** mark0n has joined #openstack-infra | 12:59 | |
| *** yfried has joined #openstack-infra | 12:59 | |
| fungi | quite a lot of "Exception while testing ssh access...Connection timed out" | 13:00 |
| *** dkehn_afk is now known as dkehn | 13:00 | |
| *** ociuhandu has quit IRC | 13:00 | |
| fungi | and also "Exception launching node id...Timeout waiting for ssh access" | 13:00 |
| *** smarcet has joined #openstack-infra | 13:00 | |
| fungi | and also quite a few nova boot attempts are resulting in servers in error state | 13:01 |
| *** ominakov has joined #openstack-infra | 13:03 | |
| *** sdake_ has quit IRC | 13:03 | |
| *** yfried__ has quit IRC | 13:04 | |
| viktors | fungi: Hi! If you have a time, could you please look at patch https://review.openstack.org/#/c/78955/ (Import oslo.db repository) ? | 13:05 |
| sdague | fungi: that ratio of deleting / building / in progress just seems like way off | 13:06 |
| sdague | unless building got very slow | 13:06 |
| fungi | sdague: yeah, i think those are nodes which are building, then the ssh test to them fails, then they're deleted, but then the providers are slow to react to delete calls | 13:07 |
| fungi | still trying to see if it's a particular provider or image or what | 13:07 |
| *** homeless has joined #openstack-infra | 13:09 | |
| fungi | yeah, i think something's up with rackspace... all the "used" state nodes are in hpcloud with the exception of a couple in rax-dfw (way too few, and nothing in use in ord or iad) | 13:09 |
| *** gokrokve has joined #openstack-infra | 13:14 | |
| *** dkranz has joined #openstack-infra | 13:16 | |
| fungi | no obvious limits maxxed out | 13:16 |
| *** unicell has quit IRC | 13:17 | |
| *** saper_ has joined #openstack-infra | 13:18 | |
| *** gokrokve has quit IRC | 13:18 | |
| *** saper has quit IRC | 13:19 | |
| *** signed8bit has joined #openstack-infra | 13:21 | |
| *** sdake_ has joined #openstack-infra | 13:21 | |
| *** doug-fish has joined #openstack-infra | 13:22 | |
| sdague | hmmmm | 13:29 |
| fungi | i'm doing some nova boot tests now just to see if i can spot anything obvious | 13:30 |
| fungi | trying with saturday's image instead of sunday too, just to rule out a new image from yesterday being the culprit | 13:30 |
| *** julim has joined #openstack-infra | 13:31 | |
| BobBall | I'm using RAX nodes just fine btw | 13:32 |
| BobBall | not that there is a huge amount of similarity | 13:32 |
| BobBall | but it's RAX and it's nodepool - that's where the similarity ends ;) | 13:32 |
| sdague | yeh, it just strikes me as odd that we can only managed 240 in use nodes | 13:32 |
| yfried | what's the difference between logs files marked as "oslofmt" to keystone log files? | 13:33 |
| yfried | https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/logstash/jenkins-log-client.yaml | 13:33 |
| fungi | took about 30 minutes to complete a bare-precise build in rax-ord, and now that it's supposedly active, i can't reach it on 22/tcp | 13:33 |
| fungi | and that was using the nodepool image from saturday | 13:34 |
| fungi | though ord is in the middle of getting upgrades for the past several days (according to support tickets and their status site), so i should test it with iad to rule out the possibility this is different | 13:35 |
| *** mriedem has joined #openstack-infra | 13:36 | |
| *** mriedem has quit IRC | 13:37 | |
| sdague | should we pull ord out then? If we're backing up on throwing out nodes and having to restart them because we allocate them in bad regions, that will definitely cause challenges. | 13:37 |
| *** mriedem has joined #openstack-infra | 13:38 | |
| fungi | sdague: no idea yet. i have a feeling what we're seeing isn't just in ord, but all of rackspace | 13:38 |
| fungi | i can get to the virtual console of the vm and it's booted to a normal login prompt, but i can't get it to respond over the network. i'll need to try building one with a script to set up an account with a known password to actually do any troubleshooting from that end though | 13:40 |
| *** lcostantino has joined #openstack-infra | 13:41 | |
| *** zns has joined #openstack-infra | 13:41 | |
| anteaya | morning | 13:43 |
| SergeyLukjanov | fungi, hey, how was your vacation? | 13:43 |
| SergeyLukjanov | anteaya, morning | 13:43 |
| anteaya | welcome back fungi | 13:43 |
| anteaya | congratulations on getting married | 13:43 |
| anteaya | w00t sdague | 13:44 |
| fungi | SergeyLukjanov: wasn't entirely vacation. more wedding and house hunting | 13:44 |
| fungi | anteaya: thanks | 13:44 |
| anteaya | welcome to your first day with the new colours | 13:44 |
| SergeyLukjanov | fungi, oh, congratulations! | 13:44 |
| sdague | fungi: are there fun pictures from the wedding some where? :) | 13:44 |
| anteaya | mostly jeblair has been streamlining activity to focus on gerrit upgrade and anything not gerrit upgrade gets put on the later list | 13:44 |
| anteaya | in terms of figuring out how many plates you want to spin this morning fungi | 13:45 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Project Groups API https://review.openstack.org/90736 | 13:45 |
| fungi | sdague: probably. christine showed me some previews the photographers put up on their facebook blog thingy, but i don't have any urls off hand. later this week probably | 13:45 |
| anteaya | yfried: the best logstash person is clarkb who is in hawaii this week | 13:45 |
| fungi | SergeyLukjanov: thanks | 13:45 |
| anteaya | some folks did take an infra off week | 13:45 |
| *** thomasem has joined #openstack-infra | 13:45 | |
| sdague | yfried: keystone used to not use oslo loging | 13:45 |
| sdague | logging | 13:46 |
| sdague | it had a custom format | 13:46 |
| anteaya | SergeyLukjanov: you are in california, are you not? | 13:46 |
| sdague | anteaya: thanks, now I just need cody to get online to bootstrap me into the network :) | 13:46 |
| anteaya | :D | 13:46 |
| anteaya | sdague: let me know if there is anything I can do to help | 13:47 |
| anteaya | I don't know cody's schedule so I have no estimate as to when he will be online | 13:47 |
| SergeyLukjanov | anteaya, not yet, I'll be in CA at this weekend | 13:47 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Project Groups API https://review.openstack.org/90736 | 13:47 |
| SergeyLukjanov | anteaya, I'm trying to finish things here and it's a bit too late I was starting doing it... | 13:48 |
| fungi | sdague: it looks like booting from snapshots is taking waaay, waaaaaaay longer than booting from base images... does that sounds likely? different storage backends maybe? | 13:48 |
| anteaya | SergeyLukjanov: ah okay | 13:48 |
| anteaya | SergeyLukjanov: /me nods | 13:48 |
| mark0n | Can anyone help me to determine if https://review.openstack.org/90092 fixes a bug? I'm not a Python programmer and thus not familiar with installation of Python modules... | 13:49 |
| anteaya | mark0n: well Marc Abramowitz has given you some feedback | 13:50 |
| mark0n | here is what happens on my Debian Wheezy machine: https://gist.github.com/mark0n/a3631044598298115358 | 13:50 |
| anteaya | any chance you can find out who this person is and ask them for some further guidance? | 13:50 |
| msabramo | mark0n: As far as I can tell that is not necessary, but maybe I am forgetting some case | 13:50 |
| anteaya | msabramo: are you Marc Abramowitz? | 13:50 |
| msabramo | anteaya, mark0n: I think we just found out who he is :-) | 13:51 |
| mark0n | anteaya: right. And what he says is right. But I still cannot _run_ JJB | 13:51 |
| msabramo | mark0n: Hmmm, let me look at your gist | 13:51 |
| msabramo | hmmm, interesting | 13:51 |
| msabramo | it seems to have downloaded the pbr egg file but then it still complains | 13:52 |
| sdague | fungi: interesting | 13:52 |
| sdague | define "way longer" | 13:52 |
| dhellmann | mark0n, msabramo, anteaya: if jjb depends on having pbr installed at runtime, that's likely a bug in jjb's requirements specifications | 13:53 |
| fungi | sdague: 3-4x | 13:53 |
| sdague | hmmm | 13:53 |
| sdague | yeh, that would explain the build time right? | 13:53 |
| *** zehicle has joined #openstack-infra | 13:53 | |
| msabramo | dhellman: oh yeah | 13:53 |
| msabramo | oops, dhellmann: oh yeah, you're right | 13:53 |
| msabramo | didn't think about that it was at runtime | 13:54 |
| fungi | sdague: though the bigger issue is the machines not being reachable over the network once booted. i can ssh into one i built from a precise base image, but not the bare-precise snapshot from saturday or sunday | 13:54 |
| msabramo | looks pretty vanilla | 13:54 |
| msabramo | setuptools.setup( | 13:54 |
| msabramo | setup_requires=['pbr'], | 13:54 |
| msabramo | pbr=True) | 13:54 |
| dhellmann | msabramo: check the requirements.txt -- the last line has pbr | 13:54 |
| anteaya | fungi: might that be a dns issue? | 13:54 |
| fungi | sdague: i probably should test taking a snapshot of a reachable server, and then deleting it and booting a new one from that | 13:55 |
| msabramo | dhellmann: ah ha! That doesn't need to be there, eh? | 13:55 |
| dhellmann | msabramo: right | 13:55 |
| fungi | anteaya: nope, this is the ip address reported by novaclient not responding over the network | 13:55 |
| dhellmann | msabramo: if you submit the patch, I can +2 :-) | 13:55 |
| dhellmann | msabramo: oh, actually, I can't on jjb :-/ | 13:55 |
| anteaya | fungi: ouch | 13:55 |
| anteaya | dhellmann: you can +1 though | 13:56 |
| msabramo | dhellmann: why does this cause the problem mark0n has though? Shouldn't setuptools still use the egg that was downloaded? | 13:56 |
| dhellmann | anteaya: yeah | 13:56 |
| dhellmann | msabramo: that's a good question | 13:56 |
| *** enikanorov__ has joined #openstack-infra | 13:56 | |
| msabramo | dhellmann: that said, this will probably fix his problem, though I'm not sure why :-) | 13:56 |
| *** zehicle_at_dell has quit IRC | 13:57 | |
| *** _nadya_ has quit IRC | 13:57 | |
| msabramo | dhellmann, mark0n: Could be a buggy version of setuptools perhaps? | 13:57 |
| dhellmann | mark0n: do you have any version of pbr installed at all? | 13:57 |
| mark0n | isn't the setup process supposed to install pbr into /usr/local/lib/python2.7/dist-packages/? | 13:57 |
| msabramo | oh maybe if he has pbr installed at the wrong version? | 13:57 |
| msabramo | maybe that overrides the egg and then it complains | 13:58 |
| dhellmann | mark0n: the install-time requirements for jjb don't specify a version, so if you have an old version installed it won't install the one that is listed as the (invalid) runtime requirement | 13:58 |
| mark0n | not before I run the commands in the Gist. I'm starting with a fresh Wheezy system | 13:58 |
| dhellmann | hmm | 13:58 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Remove pbr from requirements.txt https://review.openstack.org/90739 | 13:59 |
| dhellmann | ok, new theory: the install-time requirement for pbr isn't installed to dist-packages (because as an install-time requirement it's "temporary") | 13:59 |
| *** zehicle has quit IRC | 13:59 | |
| *** enikanorov_ has quit IRC | 13:59 | |
| dhellmann | however, since it is present while jjb itself is being installed, that temporary pbr also prevents the proper one from being installed | 13:59 |
| dhellmann | then after installation, the temporary copy is removed, leaving the system broken | 13:59 |
| msabramo | yeah I think it's something like what dhellmann is saying | 14:01 |
| msabramo | setup_requires makes it download an .egg file to the current directory | 14:01 |
| msabramo | which it uses for python setup.py stuff | 14:01 |
| msabramo | but when you run jjb, it doesn't pick that up, maybe because it's not looking in that directory | 14:01 |
| msabramo | anyway, https://review.openstack.org/90739 should fix it | 14:02 |
| fungi | sdague: off the top of your head, do you happen to know what | 14:02 |
| fungi | er | 14:02 |
| fungi | what "Instance building... 40% complete" corresponds to? | 14:02 |
| fungi | that's where nova boot seems to spend the majority of its time | 14:03 |
| mark0n | msabramo: I can confirm that this fixes the issue! | 14:03 |
| msabramo | mark0n: Awesome! Thanks for verifying! | 14:04 |
| msabramo | mark0n: Give it a +1 at https://review.openstack.org/90739 | 14:04 |
| mark0n | msabramo: done | 14:04 |
| sdague | fungi: nope | 14:04 |
| sdague | that on rax? | 14:04 |
| fungi | sdague: yep | 14:05 |
| chmouel | it would be nice to have this plugin installed for that shiny new 2.8 install we will have https://gerrit.googlesource.com/plugins/reviewers-by-blame/ | 14:06 |
| fungi | sdague: bingo! | 14:07 |
| fungi | sdague: ip address bound to eth0 is *not* the ip address reported by nova | 14:07 |
| *** rwsu has joined #openstack-infra | 14:08 | |
| anteaya | chmouel: hey | 14:09 |
| anteaya | chmouel: your review tools ux session got accepted, congratulations | 14:09 |
| fungi | the ip address reported bound to eth0 seems to correspond to an iface eth0 inet static stanza in /etc/network/interfaces. wonder how that's getting there (cloud-init?) | 14:09 |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/requirements: Bump python-keystoneclient to >= 0.8.0 https://review.openstack.org/90745 | 14:10 |
| anteaya | chmouel: it is a bit late in the process for us to consider new gerrit plugins | 14:10 |
| anteaya | since we have had no time to test them | 14:10 |
| anteaya | I can suggest that jim take a look at it | 14:10 |
| anteaya | but I can't make any promises | 14:10 |
| anteaya | chmouel: thanks for letting us know about it though | 14:10 |
| *** ociuhandu has joined #openstack-infra | 14:11 | |
| fungi | anteaya: chmouel: once we're done with the upgrade, we can certainly test new plugins on review-dev (maybe next week?) and evaluate whether they're worthwhile to add | 14:11 |
| anteaya | there we go, that sounds like a plan | 14:11 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to 0.13 https://review.openstack.org/90746 | 14:11 |
| *** jgrimm has joined #openstack-infra | 14:13 | |
| Shrews | fungi! wb & congrats! | 14:14 |
| *** andreaf has quit IRC | 14:14 | |
| fungi | Shrews: thanks | 14:14 |
| Shrews | fungi: olaph and i have decided you must meet us at Dos Perros before you move | 14:14 |
| *** gokrokve has joined #openstack-infra | 14:14 | |
| fungi | Shrews: i've still got about 6 weeks, so can definitely work that in | 14:14 |
| Shrews | or, anywhere, really | 14:14 |
| fungi | and then you'll have to come meet me at the beach after i move ;) | 14:15 |
| mordred | look, it's a fungi! | 14:15 |
| Shrews | fungi: cool. as soon as olaph quits tending his goats, chickens, cows and ducks and makes his way back to irc, we'll work out something | 14:16 |
| fungi | mordred: what did i miss of note? gerrit upgrade slated for an hour and change from now... did we also change how puppet agent is being triggered? | 14:17 |
| fungi | though i saw some mention of that in scrollback | 14:17 |
| fungi | er, thought | 14:17 |
| *** msabramo has quit IRC | 14:18 | |
| anteaya | fungi: yes | 14:18 |
| mordred | fungi: yup | 14:18 |
| anteaya | it is being triggered via ssh | 14:18 |
| mordred | what anteaya said | 14:18 |
| anteaya | hey mordred | 14:18 |
| mordred | hey anteaya | 14:18 |
| anteaya | I have to be afk for about 20 minutes | 14:19 |
| *** mark0n has quit IRC | 14:19 | |
| anteaya | back soon | 14:19 |
| *** gokrokve has quit IRC | 14:19 | |
| mordred | fungi: also, I've got a patch that's ready for review around using diskimage-builder in nodepool | 14:19 |
| *** ociuhandu has quit IRC | 14:19 | |
| mordred | fungi: and by review, I mean it should be possible for you to pull the patch, follow instructions ina readme and produce a diskimage locally | 14:19 |
| mordred | which you can then mount or whatever and inspect | 14:20 |
| fungi | mordred: exciting. right now i'm just trying to figure out why rackspace nodepool servers have the wrong ip addresses for eth0 in /etc/network/interfaces | 14:20 |
| *** exz has quit IRC | 14:20 | |
| mordred | fungi: wow. that's exciting | 14:20 |
| fungi | did we merge any changes which could have caused that, maybe fridayish? | 14:20 |
| mordred | I don't _think_ we did anything related to /etc/network/interfaces | 14:20 |
| *** exz has joined #openstack-infra | 14:21 | |
| fungi | i'm booting a second one now to see if maybe they're all inheriting an ip address from the snapshot image | 14:21 |
| *** _nadya_ has joined #openstack-infra | 14:21 | |
| fungi | or whether something else (file injection? cloud-init?) is just doing the wrong thing | 14:21 |
| *** skolekonov has left #openstack-infra | 14:22 | |
| *** gokrokve has joined #openstack-infra | 14:22 | |
| *** rcarrillocruz has joined #openstack-infra | 14:22 | |
| sdague | fungi: I looked through devstack logs and didn't find anything suspect | 14:23 |
| fungi | sdague: well, the one i'm testing with is the bare-precise snapshot, so devstack wouldn't have anything to do with that | 14:23 |
| BobBall | fungi/mordred: Any chance one of you could approve an urgent* fix for devstack at https://review.openstack.org/#/c/90660/? *the xenserver 3rd party CI is broken waiting on this change | 14:24 |
| mordred | dhellmann, anteaya: fyi- there is a bug in setuptools that was being tripped earlier | 14:24 |
| mordred | the fix is to use "pip install ." instead of python setup.py install | 14:24 |
| *** rcarrill` has quit IRC | 14:24 | |
| sdague | BobBall: I can push it in | 14:24 |
| BobBall | Ah - great, thanks sdague! as you can see, tests all came back with green things. | 14:25 |
| *** adalbas has quit IRC | 14:25 | |
| jeblair | fungi: good morning, congratulations, and welcome back! | 14:25 |
| *** annegentle has joined #openstack-infra | 14:25 | |
| fungi | jeblair: thanks | 14:25 |
| mordred | morning jeblair | 14:26 |
| mordred | fungi: and yes, congratulations! | 14:26 |
| jeblair | mordred: good morning | 14:26 |
| fungi | thanks mordred | 14:26 |
| jeblair | i see fungi has been briefed | 14:26 |
| fungi | briefly briefed anyway | 14:27 |
| jeblair | briefly briefed is the best brand of briefed | 14:27 |
| *** pcrews has joined #openstack-infra | 14:28 | |
| fungi | okay, situation confirmed. building two virtual machines from the same nodepool snapshot, the're coming up with the same (wrong) ip addresses bound to eth0, which match what's statically set in /etc/network/interfaces | 14:29 |
| dhellmann | mordred: nice, I hadn't spotted that mark0n was using setup.py install | 14:29 |
| *** gokrokve has quit IRC | 14:29 | |
| jeblair | #status notice Gerrit downtime for upgrade begins in 90 minutes. See: https://wiki.openstack.org/wiki/GerritUpgrade | 14:30 |
| openstackstatus | jeblair: sending notice | 14:30 |
| mordred | jeblair: any reason we don't use list_images.sh in cache_devstack.py ? | 14:30 |
| fungi | how do we expect rackspace to normally set ip addresses? dhcp? | 14:30 |
| *** david-lyle has joined #openstack-infra | 14:30 | |
| mordred | dhellmann: I would love it if setuptools worked, btw | 14:30 |
| *** david-lyle has quit IRC | 14:30 | |
| dhellmann | mordred: join the club | 14:30 |
| jeblair | mordred: only a matter of time. there may even be a review for that. unsure. | 14:30 |
| mordred | jeblair: ok. so it's a thing we think is a good idea | 14:30 |
| -openstackstatus- NOTICE: Gerrit downtime for upgrade begins in 90 minutes. See: https://wiki.openstack.org/wiki/GerritUpgrade | 14:30 | |
| jeblair | fungi: file injection of /etc/network/interfaces i believe. | 14:30 |
| *** david-lyle has joined #openstack-infra | 14:30 | |
| jeblair | mordred: very much so, i believe it was written for exactly that purpose | 14:31 |
| fungi | jeblair: okay, seems to be busted in that case | 14:31 |
| mordred | jeblair: great. | 14:31 |
| dhellmann | mordred: to be fair, we do tend to go out to the rough edges for a lot of cases | 14:31 |
| mordred | dhellmann: :) | 14:31 |
| *** david-lyle has quit IRC | 14:31 | |
| openstackstatus | jeblair: finished sending notice | 14:31 |
| *** david-lyle has joined #openstack-infra | 14:32 | |
| *** mdenny has joined #openstack-infra | 14:32 | |
| jeblair | fungi: has the problem you are working on broken all our nodepool rax nodes? | 14:32 |
| fungi | jeblair: yes | 14:33 |
| fungi | jeblair: seems that way | 14:33 |
| jeblair | fungi: have you spun up a node from a base image? | 14:33 |
| fungi | jeblair: just trying to get some confirmation before i open a support case with them, | 14:33 |
| fungi | jeblair: i have, and it works fine. doing another now | 14:33 |
| jeblair | fungi: in those cases, does it appear it got the ip from file injection or dhcp (or some other magic?) | 14:34 |
| fungi | jeblair: that's what i'm checking with the new one i'm spinning up | 14:34 |
| fungi | should know in a couple more minutes | 14:34 |
| * jeblair feels like he is nearly caught up. :) | 14:34 | |
| *** rfolco_ has quit IRC | 14:34 | |
| jeblair | fungi: how old are our nodepool rax images? | 14:35 |
| fungi | jeblair: saturday and sunday | 14:35 |
| *** annegentle has quit IRC | 14:35 | |
| jeblair | k | 14:35 |
| fungi | jeblair: i've been testing with an image from saturday just to see if it was any more likely to work, but no such luck | 14:35 |
| jeblair | fungi: i believe on friday we merged a change to chattr +i /etc/resolv.conf | 14:36 |
| fungi | perhaps that breaks all their file injection when it can't overwrite that | 14:36 |
| jeblair | fungi: because apparently, somehow the rackspace images were managing to overwrite them; i was suspecting that they may have been directly injecting that too | 14:36 |
| jeblair | fungi: that's what i was just starting to think | 14:36 |
| anteaya | back | 14:37 |
| jeblair | if so, then we have a choice: dns or network | 14:37 |
| pleia2 | going to tweet about the upgrade one last time, all still on track? (I'm on vacation, but figured I'd pop in) | 14:37 |
| fungi | jeblair: gigantic hosts files ftw! | 14:37 |
| jeblair | pleia2: yep | 14:37 |
| pleia2 | cool, good luck :) | 14:37 |
| jeblair | mordred: we may want to think about either doing something in rc.local to fix resolv, or having nodepool do it | 14:37 |
| jeblair | but also.... | 14:37 |
| mattymo | gerrit is taking for ever to git review (thankfully the upgrade will start soon and solve all problems!) | 14:38 |
| mordred | jeblair: I have a think in rc.local in the nodepool patch | 14:38 |
| mordred | jeblair: I could break that out and have puppet do it | 14:38 |
| fungi | jeblair: yep, confirmed. working server booted from a base image instead of a snapshot has a (correct) static stanza in the interfaces file, so they do indeed seem to be trying to inject that | 14:38 |
| *** sandywalsh has joined #openstack-infra | 14:38 | |
| jeblair | mordred: but also, i still want to know how we will have our image configure network -- eg, will we need to do file injection on rax and dhcp on hp? etc... | 14:38 |
| jeblair | mordred: (with dib) | 14:38 |
| jeblair | fungi: let's revert the chattr +i change and see if it fixes things | 14:39 |
| fungi | jeblair: agreed. we'll need new rackspace images after that | 14:39 |
| jeblair | yup | 14:39 |
| mordred | jeblair: https://review.openstack.org/#/c/88479/11/modules/openstack_project/files/nodepool/elements/fix-bad-clouds/finalise.d/99-unbound | 14:39 |
| mordred | line 36 | 14:39 |
| mordred | is what I'm having the rc.local file do | 14:39 |
| *** exz has quit IRC | 14:39 | |
| *** exz has joined #openstack-infra | 14:40 | |
| *** wenlock has joined #openstack-infra | 14:40 | |
| mordred | jeblair: good question re: images and network - I think it should Just Work though - rax can file inject /etc/network/interfaces on rackspace | 14:40 |
| jeblair | mordred: that's not the worst thing in the world. | 14:40 |
| mordred | regardless of what our image content it | 14:40 |
| mordred | is | 14:40 |
| jeblair | mordred: but how will it know to do that? | 14:40 |
| mordred | jeblair: I don't think the image has to know anything | 14:41 |
| mordred | I think that's a reach-down-from-outside thing rackspace just does? | 14:41 |
| *** mrodden has joined #openstack-infra | 14:41 | |
| jeblair | mordred: how will rax know to write that file? that would be useless on redhat. | 14:41 |
| *** adalbas has joined #openstack-infra | 14:41 | |
| mordred | hrm | 14:41 |
| mordred | I honestly have no idea how their fileinjection works | 14:41 |
| *** prad_ has joined #openstack-infra | 14:41 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Replace raw.github.com with raw.githubusercontent.com https://review.openstack.org/90757 | 14:42 |
| jeblair | right, this is the other prong of the dib problem i was hoping someone would work on in parallel with you | 14:42 |
| mordred | maybe they have scripts that looks at the host and determines, rh vs. deb vs. windows/ | 14:42 |
| jeblair | what would be much nicer is if, say, rax supported dhcp behind the scenes and we could just dispense with file injection on our images | 14:42 |
| derekh | I think nodepool templates are failing to build because of ^ | 14:42 |
| mordred | jeblair: ++ | 14:43 |
| *** mwagner_lap has quit IRC | 14:43 | |
| *** exz has quit IRC | 14:43 | |
| *** thomasbiege has joined #openstack-infra | 14:43 | |
| jeblair | mordred: i love how we still depend on github. | 14:43 |
| mordred | derekh: +2 | 14:43 |
| mordred | jeblair: we could, you know download that file and host it on one of our servers | 14:43 |
| *** exz has joined #openstack-infra | 14:44 | |
| jeblair | mordred: i assumed we wanted the latest version or something | 14:44 |
| mordred | jeblair: in general, yes. but we do cache pip releases in pypi.o.o | 14:44 |
| jeblair | derekh: aprvd | 14:44 |
| mordred | jeblair: so we could just cache a version of the script, reference that in our stuff, and have it install pip from our mirror | 14:44 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Revert "Revert "update grenade branches"" https://review.openstack.org/90043 | 14:45 |
| mordred | jeblair: you want me to cook up a puppet version of the rc.local patch? | 14:45 |
| sdague | oh finally ^^^^ Revert "Revert "update grenade branches"" https://review.openstack.org/90043 | 14:45 |
| anteaya | sdague: \o/ | 14:46 |
| anteaya | it was quite the bug parade | 14:46 |
| sdague | you should look at the recheck comment list on that review, I managed to uncover 3 previously unknown races in it | 14:46 |
| sdague | mostly nova issues | 14:46 |
| jeblair | mordred: i'm actually wondering if it would be better to leave that in nodepool | 14:46 |
| jeblair | mordred: we shouldn't need it on long running servers | 14:46 |
| mordred | jeblair: nod. good point | 14:47 |
| mordred | prepare_node.sh then? | 14:47 |
| jeblair | mordred: yeah | 14:47 |
| mordred | we already do unbound stuffs there | 14:47 |
| jeblair | mordred: and with that in place, all the chattr stuff can go away i believe | 14:47 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Revert "Always make /etc/resolv.conf immutable" https://review.openstack.org/90761 | 14:47 |
| *** talluri has joined #openstack-infra | 14:47 | |
| jeblair | hrm | 14:47 |
| jeblair | mordred: actually it may still be necessary on redhat | 14:47 |
| mordred | really? | 14:47 |
| mordred | whyso? | 14:48 |
| jeblair | mordred: that's why we put it there in the first place | 14:48 |
| anteaya | jeblair: chmouel brought this gerrit plugin to our attention: https://gerrit.googlesource.com/plugins/reviewers-by-blame/ fungi suggests we might be able to test it on review-dev next week and then go from there | 14:48 |
| *** salv-orlando has joined #openstack-infra | 14:48 | |
| jeblair | mordred: because all of the places where we could disable the write-resolv-conf behavior are file-injected on rax | 14:48 |
| *** talluri has quit IRC | 14:49 | |
| jeblair | mordred: so actually, chattr probably can go away with the rc.local change | 14:49 |
| *** rcarrill` has joined #openstack-infra | 14:49 | |
| *** talluri has joined #openstack-infra | 14:49 | |
| jeblair | mordred: since it will fix the nodepool case, and the long-running-server case should similarly self-correct | 14:50 |
| fungi | jeblair: do you still have an e-mail discussion thread going with rax engineering about the dns blacklist lag? would be nice if they could suggest a working solution to not using their resolvers but still having working network interfaces | 14:50 |
| jeblair | fungi: it died off | 14:50 |
| *** rcarrillocruz has quit IRC | 14:50 | |
| jeblair | fungi: i think i'd like to just hold off and see if we can use our own images to fix this | 14:51 |
| fungi | i guess rc.local is a viable solution so long as we don't need dns resolution prior to rc.local being run | 14:51 |
| jeblair | fungi: and if it becomes a problem then, continue the discussion | 14:51 |
| fungi | makes sense | 14:51 |
| mordred | ++ | 14:51 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Overrwrite resolve.conf at boot time https://review.openstack.org/90763 | 14:51 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Remove chattr https://review.openstack.org/90764 | 14:51 |
| *** talluri_ has joined #openstack-infra | 14:51 | |
| *** talluri_ has quit IRC | 14:52 | |
| *** talluri has quit IRC | 14:52 | |
| *** atiwari has joined #openstack-infra | 14:52 | |
| fungi | so we want to try that stack as a fix? if so i'll abandon the revert in 90761 | 14:53 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch review.o.o out of testmode https://review.openstack.org/90767 | 14:53 |
| *** talluri has joined #openstack-infra | 14:53 | |
| zaro | morning | 14:53 |
| *** talluri has quit IRC | 14:54 | |
| anteaya | morning zaro | 14:54 |
| jeblair | fungi: yeah, i think we could try those 2 instead | 14:55 |
| jeblair | fungi: i +2d; feel free to aprv | 14:55 |
| jeblair | zaro: morning | 14:56 |
| mordred | morning zaro | 14:56 |
| *** gokrokve has joined #openstack-infra | 14:56 | |
| jeblair | mordred: did your 'how to stop puppet' doc change land? | 14:56 |
| *** _nadya_ has quit IRC | 14:57 | |
| fungi | mordred: does the empty line before the shebang in 90763 do something special? | 14:57 |
| *** gokrokve has quit IRC | 14:57 | |
| *** gokrokve has joined #openstack-infra | 14:58 | |
| *** gokrokve has quit IRC | 14:58 | |
| *** sandywalsh has quit IRC | 15:01 | |
| *** Longgeek_ has joined #openstack-infra | 15:01 | |
| jeblair | yes, the doc change did merge but it's not reflected at http://ci.openstack.org/puppet.html | 15:02 |
| *** sandywalsh has joined #openstack-infra | 15:02 | |
| mordred | weird | 15:02 |
| mordred | well - simple story "puppet agent --disable" | 15:02 |
| *** Ryan_Lane has joined #openstack-infra | 15:03 | |
| jeblair | i'm guessing the docs build or publish job is broken | 15:03 |
| mordred | and then the thing in the docs that tells you how to run it by hand | 15:03 |
| *** chandan_kumar has quit IRC | 15:03 | |
| jeblair | fungi, mordred: upgrade plan here: https://etherpad.openstack.org/p/J8WNqyP2dC | 15:03 |
| *** Longgeek has quit IRC | 15:03 | |
| *** rcarrillocruz has joined #openstack-infra | 15:04 | |
| *** beekneemech is now known as bnemec | 15:04 | |
| anteaya | how far to run? | 15:04 |
| fungi | jeblair: the upgrade will involve replacing review.o.o with a new server? | 15:04 |
| mordred | fungi: yes | 15:04 |
| fungi | i guess so. my connection lagged | 15:05 |
| jeblair | fungi: yes! i guess that was left out of your briefing! :) | 15:05 |
| mordred | fungi: the new server is a performance node - we tried upgrading to trusty too - but that's a bucket of fail right now, so we have to wait for puppet3-ification everywhere | 15:05 |
| fungi | jeblair: it was merely left out of my reading comprehension going through the ml posts and meeting minutes, more likely | 15:05 |
| *** rcarrill` has quit IRC | 15:05 | |
| jeblair | fungi: shiny new performance2 node. /home/gerrit2 is on a center volume. | 15:05 |
| mordred | or cinder | 15:05 |
| fungi | keen | 15:05 |
| jeblair | that too | 15:06 |
| jeblair | fungi: that tidbit might have just been buried in channel logs. | 15:06 |
| fungi | is old review.o.o going git replication to new review.o.o or is there an rsync step hidden in there? | 15:06 |
| jeblair | fungi: rsync | 15:07 |
| jeblair | fungi: also, db is trove | 15:07 |
| fungi | oh, nice! | 15:07 |
| mordred | which, I believe, makes paste our last non-trove thing | 15:07 |
| mordred | well, and puppetdb which runs postgres | 15:08 |
| jeblair | fungi: we moved the local apache git mirror and the jeepyb git cache to /opt/lib (from /var/lib) so that they live on the ephemeral volume | 15:09 |
| *** yfried has quit IRC | 15:12 | |
| *** Ryan_Lane has quit IRC | 15:13 | |
| zaro | for the upgrade today, I was wondering who will be running the db upgrade? | 15:14 |
| jeblair | zaro: one of mordred, fungi, or i | 15:14 |
| *** Ryan_Lane has joined #openstack-infra | 15:14 | |
| jeblair | mordred: can you fix https://review.openstack.org/#/c/90763/1 before we shut gerrit down? | 15:16 |
| mordred | jeblair: yes | 15:16 |
| fungi | mordred: did you see my question above for 90763? (soft -1 since it's wrong-but-not-broken and i'd love to get that merged before we shut down gerrit) | 15:17 |
| *** dizquierdo has quit IRC | 15:17 | |
| jeblair | we can have image builds running once that merges | 15:17 |
| fungi | oh, what jeblair said | 15:17 |
| anteaya | what will zuul do when gerrit is down? | 15:18 |
| fungi | anteaya: a lot of nothing | 15:18 |
| anteaya | okay | 15:18 |
| *** marun has joined #openstack-infra | 15:18 | |
| anteaya | so how will patches being tested get their reports? | 15:19 |
| anteaya | or will zuul just pick them up again later | 15:19 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Overrwrite resolve.conf at boot time https://review.openstack.org/90763 | 15:19 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Remove chattr https://review.openstack.org/90764 | 15:19 |
| *** talluri has joined #openstack-infra | 15:19 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Cleanup hacking doc generation https://review.openstack.org/90483 | 15:19 |
| anteaya | I'm wondering if I should be advising people to recheck no bug if they are in the queue when gerrit goes offline | 15:20 |
| jeblair | anteaya: we could snapshot the queue when we shut down gerrit | 15:20 |
| fungi | i assume the plan is to dump zuul's queues and then kill it and restore later | 15:20 |
| anteaya | great | 15:20 |
| fungi | yeah, that | 15:20 |
| jeblair | it is now :) | 15:20 |
| anteaya | :D | 15:20 |
| anteaya | thanks | 15:20 |
| fungi | heh | 15:20 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix minor non-important error in cache_devstack https://review.openstack.org/90775 | 15:20 |
| jeblair | i've stopped puppet on both review hosts; that should mean it's safe to merge all the gerrit upgrade changes | 15:20 |
| *** Ryan_Lane has quit IRC | 15:21 | |
| mordred | jeblair: awesome | 15:21 |
| jeblair | fungi: would you care to get started on that? | 15:21 |
| jeblair | s/stopped puppet/disabled puppet/ | 15:21 |
| fungi | jeblair: get started merging the gerrit changes? sure | 15:22 |
| jeblair | fungi: looking at the zuul queue, i'm not sure they will make it through in time. i think we may want to bypass zuul on these. | 15:22 |
| *** talluri has quit IRC | 15:23 | |
| fungi | jeblair: i can enqueue them | 15:23 |
| *** talluri has joined #openstack-infra | 15:24 | |
| jeblair | fungi: i was suggesting that you join project bootstrappers and force-merge them | 15:24 |
| *** ildikov_ has quit IRC | 15:24 | |
| *** talluri has quit IRC | 15:24 | |
| fungi | could do that too. need to be in that group to add the new branch anyway | 15:24 |
| *** viktors is now known as viktors|afk | 15:24 | |
| jeblair | new branch? | 15:25 |
| mordred | jeblair: further down in the etherpad is a list of existing changes | 15:25 |
| mordred | gah | 15:25 |
| mordred | fungi: ^^ | 15:25 |
| fungi | line 7 of https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes | 15:25 |
| fungi | or has that already been done? | 15:25 |
| *** thomasbiege has quit IRC | 15:26 | |
| jeblair | fungi: oh that's done; whoever did that just didn't mark it | 15:26 |
| fungi | gah. okay, so the ones which say "approve after upgrade" should actually be approved now before the upgrade? | 15:26 |
| *** talluri has joined #openstack-infra | 15:27 | |
| jeblair | fungi: yep. i have no idea how we would approve them after the upgrade :) | 15:27 |
| mordred | jeblair, fungi: do you guys have any objection to a) installing vim-nox on new review b) adding it to the base puppet manifest? I have way too many muscle memories for typing "vim $blah" | 15:27 |
| fungi | jeblair: i tend to agree ;) | 15:27 |
| jeblair | mordred: that's fine; it used to come installed on the rax image i think, but not on the pvhvm ones i guess | 15:28 |
| fungi | mordred: no objection. seems like something which was stripped out of the new pvhvm images | 15:28 |
| mordred | kk | 15:28 |
| fungi | that | 15:28 |
| *** e0ne has quit IRC | 15:28 | |
| *** talluri has quit IRC | 15:28 | |
| *** e0ne has joined #openstack-infra | 15:29 | |
| jeblair | there's a lot of repetition in these upgrade steps, i'm trying to sort it out | 15:29 |
| anteaya | I'm standing by, let me know if there is something I can do to help | 15:29 |
| jeblair | #status notice Gerrit downtime for upgrade begins in 30 minutes. See: https://wiki.openstack.org/wiki/GerritUpgrade | 15:29 |
| openstackstatus | jeblair: sending notice | 15:29 |
| *** dizquierdo has joined #openstack-infra | 15:30 | |
| *** markmcclain has joined #openstack-infra | 15:30 | |
| jeblair | zaro: why is "# Enable copying of vote scores: https://review.openstack.org/#/c/69800" listed in https://etherpad.openstack.org/p/gerrit_upgrade_script and not in https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes ? | 15:30 |
| -openstackstatus- NOTICE: Gerrit downtime for upgrade begins in 30 minutes. See: https://wiki.openstack.org/wiki/GerritUpgrade | 15:30 | |
| fungi | oh, crikey... merge conflicts | 15:31 |
| mordred | fungi: want help? | 15:31 |
| openstackstatus | jeblair: finished sending notice | 15:31 |
| jeblair | zaro: i moved it | 15:31 |
| *** talluri has joined #openstack-infra | 15:32 | |
| fungi | mordred: jeblair: zaro: a lot of the changes linked from the end of https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes (lines 30-38) seem to have merge conflicts, negative votes, and so on | 15:32 |
| jeblair | zaro: do you have a copy of the all-projects acl config as it should appear in production after the upgrade? i just see steps about how to change it. | 15:32 |
| fungi | i'll start trying to tackle them in order | 15:33 |
| jeblair | zaro: if not, would you please prepare one that we can simply install? | 15:33 |
| *** e0ne has quit IRC | 15:33 | |
| zaro | jeblair: it's on the remaining-gerrit-upgrade-changes etherpad | 15:33 |
| zaro | jeblair: manually configure review.o.o | 15:33 |
| zaro | jeblair: don't understand what you mean about 69800, it looks like it already merged. | 15:34 |
| jeblair | zaro: that's a change, and it doesn't change anything called "All-Projects.config" which means it's not what we need | 15:34 |
| YorikSar | zaro, jeblair: Hi. Did you consider my proposal to make -1: WIP autocopied to new patchsets in our new Gerrit? ML thread seems to be ended with a question... | 15:34 |
| mordred | jeblair: I think he means it's in https://review.openstack.org/#/c/70818/10/doc/source/gerrit.rst | 15:35 |
| jeblair | YorikSar: this is not a good time to have that conversation | 15:35 |
| mordred | jeblair: the new version of the docs acl file has been updated | 15:35 |
| zaro | jeblair: it's just a doc change that shows how the acls should be configured. | 15:35 |
| jeblair | zaro: can you extract that into a paste that we can just copy/paste? | 15:35 |
| zaro | jeblair: the change for gerrit.rst | 15:35 |
| YorikSar | jeblair: Ok | 15:35 |
| *** talluri has quit IRC | 15:35 | |
| zaro | jeblair: will do. | 15:35 |
| jeblair | zaro: thanks. as for https://review.openstack.org/#/c/69800 i haven't looked at the change, i just saw that the upgrade script said to merge it, so i asked about it | 15:36 |
| jeblair | zaro: we can't actually merge any changes while we're running the upgrade script, so i moved it to the etherpad of changes that need to merge | 15:36 |
| jeblair | zaro: oh, that change only applies to review-dev. is there an equivalent change for review? | 15:37 |
| *** talluri has joined #openstack-infra | 15:37 | |
| mordred | zaro, jeblair: I can get the all-projects acl step - | 15:38 |
| BobBall | What'll happen with jobs in the gate queue when gerrit goes bye bye? | 15:39 |
| jeblair | i'm going to start the rsync's | 15:39 |
| *** Longgeek_ has quit IRC | 15:39 | |
| anteaya | BobBall: we will snapshot them | 15:41 |
| zaro | jeblair: the equivalent change for review is 82416 and it's on the list to approve after upgrade. | 15:41 |
| anteaya | once new gerrit is up, zuul should pick up the snapshotted queue | 15:41 |
| anteaya | BobBall: if that fails we will have suggestions | 15:41 |
| BobBall | fair enough :) | 15:41 |
| jeblair | zaro: cool, thanks | 15:41 |
| anteaya | BobBall: thanks | 15:41 |
| zaro | mordred: i added all-project.config setting to the bottom of the etherpad. | 15:42 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 15:42 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Activate secondary index support for Gerrit https://review.openstack.org/79005 | 15:42 |
| *** pblaho has quit IRC | 15:42 | |
| fungi | zaro: sanity check on those rebases? ^ | 15:42 |
| jeblair | i'm manually editing sshd_config on old-review to allow root logins without-password to do the rsync | 15:43 |
| *** gokrokve has joined #openstack-infra | 15:44 | |
| *** asselin__ has joined #openstack-infra | 15:45 | |
| *** ociuhandu has joined #openstack-infra | 15:45 | |
| *** atiwari_ has joined #openstack-infra | 15:45 | |
| anteaya | oh in case we have a net split, is everyone also on oftc? | 15:45 |
| *** stevebaker has quit IRC | 15:45 | |
| *** asselin_ has quit IRC | 15:46 | |
| *** atiwari has quit IRC | 15:46 | |
| *** dangers has joined #openstack-infra | 15:46 | |
| fungi | anteaya: yes | 15:46 |
| anteaya | fungi: great | 15:46 |
| fungi | (i am anyway) | 15:46 |
| anteaya | I see you | 15:46 |
| *** dangers_away has quit IRC | 15:46 | |
| anteaya | don't know if it is a concern or not, hopefully not | 15:47 |
| jeblair | anteaya: i was not, thanks for the reminder | 15:47 |
| anteaya | np | 15:47 |
| jeblair | (am now) | 15:47 |
| anteaya | zaro? you on oftc #openstack-infra | 15:47 |
| *** markmcclain has quit IRC | 15:47 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Move git checks into their own file https://review.openstack.org/90265 | 15:47 |
| zaro | anteaya: no i'm not | 15:47 |
| fungi | jeblair: "merge all the changes" might also need puppet disabled on more than review.o.o, yeah? for example thinking https://review.openstack.org/#/c/79103/7/modules/openstack_project/files/zuul/layout.yaml could be disruptive | 15:47 |
| anteaya | zaro: can you join easily? | 15:48 |
| mordred | jeblair, zaro: http://paste.openstack.org/show/77464/ | 15:48 |
| mordred | I've gota n all-projects git checked out locally with that patch applied | 15:48 |
| anteaya | mordred: you are coming too, right? | 15:48 |
| zaro | anteaya: don't know how | 15:48 |
| anteaya | zaro: k | 15:48 |
| jeblair | fungi: i'll disable puppet on zuul | 15:48 |
| zaro | jeblair, fungi :sanity checking. | 15:48 |
| *** gokrokve has quit IRC | 15:48 | |
| fungi | zaro: thanks | 15:48 |
| fungi | jeblair: okay, i'll go ahead with zuul-affecting config changes too in that case | 15:49 |
| jeblair | fungi: done | 15:49 |
| anteaya | zaro: http://www.oftc.net/ | 15:49 |
| anteaya | if we net split | 15:49 |
| anteaya | otherwise don't worry about it | 15:49 |
| openstackgerrit | A change was merged to openstack-infra/config: update trigger to match new workflow label https://review.openstack.org/79103 | 15:50 |
| jeblair | rsync -a --delete root@198.101.231.251:/home/gerrit2/review_site/ /home/gerrit2/review_site | 15:50 |
| jeblair | that look about right ^ ? | 15:50 |
| jeblair | oops | 15:50 |
| jeblair | rsync -a --delete root@198.101.231.251:/home/gerrit2/review_site/ /home/gerrit2/review_site/ | 15:50 |
| *** jaypipes has quit IRC | 15:51 | |
| zaro | mordred: ALCs LGTM | 15:51 |
| *** stevebaker has joined #openstack-infra | 15:51 | |
| *** mwagner_lap has joined #openstack-infra | 15:51 | |
| jeblair | Gerrit is unavailable until further notice for a major upgrade. See: https://wiki.openstack.org/wiki/GerritUpgrade | 15:54 |
| jeblair | anteaya: ^ look good for the alert announcement? (will be the channel topic everywhere) | 15:54 |
| *** gyee has joined #openstack-infra | 15:54 | |
| mordred | jeblair: yes | 15:54 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o https://review.openstack.org/82416 | 15:54 |
| fungi | zaro: and that rebase too ^ | 15:55 |
| anteaya | jeblair: yes | 15:55 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: upgrade review.o.o to gerrit ver 2.8.4 https://review.openstack.org/81622 | 15:55 |
| mordred | fungi: lgtm | 15:56 |
| zaro | new change for approve after upgrade ^ | 15:56 |
| *** adalbas has quit IRC | 15:56 | |
| fungi | 79107 is going to be a fun, fun, fun rebase | 15:56 |
| mordred | fungi: seems like a good candidate for just pulling master and running a script on | 15:57 |
| fungi | mordred: maybe... it's also adding acls for some projects (accidentally because of a prior broken rebase?) | 15:58 |
| fungi | zaro: ^ ? | 15:58 |
| mordred | you know what? perhaps in the future with bulk changes like that- we should also post the script we used to generate them in case of need to re-do | 15:59 |
| mordred | fungi: well - the main repeated things seem to be: | 15:59 |
| jeblair | and/or keep them up to date | 15:59 |
| mordred | fungi: s/Approved = +0..+1/Workflow = -1..+1/ ; d/^.*workInProgress =.*$/ | 15:59 |
| jeblair | initial rsyncs are complete | 15:59 |
| *** jgallard has quit IRC | 15:59 | |
| mordred | fungi: and s/label-Approved/label-Workflow/ | 16:00 |
| zaro | fungi: ohh sorry about that one, probably should have done that yesterday. | 16:00 |
| zaro | fungi: probably better just to start over there. | 16:01 |
| *** dkliban is now known as dkliban_afk | 16:01 | |
| mordred | jeblair: yah - keeping up to date is good - but if someone is following us with their own system, scripts we used to generate a particular commit might be helpful either as docs for what changed or to use (also, reading the script might be an easier way for us to review the intent of the change_ | 16:02 |
| mordred | just a thought for next time | 16:02 |
| anteaya | 16:02 utc | 16:02 |
| jeblair | mordred: ++ | 16:02 |
| jeblair | anteaya: yeah, we're waiting on those changes to be fixed/reviewed/merged | 16:02 |
| * anteaya nods | 16:02 | |
| jeblair | but yes, we are now starting late :( | 16:03 |
| *** sweston has joined #openstack-infra | 16:03 | |
| anteaya | no no | 16:03 |
| jeblair | i don't think it will be late enough to warrant an announcement | 16:03 |
| fungi | zaro: sed -e s/label-Approved/label-Workflow/ -e s/Workflow = +0..+1/Workflow = -1..+1/ -e /.*workInProgress.*/d | 16:03 |
| anteaya | just wanted to ensure we hadn't lost track is all | 16:03 |
| zaro | ok, i think i've sanity checked all so far. | 16:04 |
| anteaya | not rushing | 16:04 |
| anteaya | sweston: hey | 16:04 |
| mordred | fungi: yes. looks right to me | 16:04 |
| anteaya | sweston: https://etherpad.openstack.org/p/J8WNqyP2dC and https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes and https://etherpad.openstack.org/p/gerrit_upgrade_script | 16:04 |
| *** jaypipes has joined #openstack-infra | 16:05 | |
| anteaya | sweston: and if it is easy to join http://www.oftc.net/ #openstack-infra in case of a net split, that is great too but low priority | 16:05 |
| *** marcoemorais has joined #openstack-infra | 16:05 | |
| zaro | fungi: sorry, i no sed afficianado. | 16:05 |
| SergeyLukjanov | jeblair, mordred, fungi, zaro is it ok that review.o.o is still working? | 16:06 |
| anteaya | SergeyLukjanov: yes | 16:06 |
| mordred | SergeyLukjanov: yes. we have not turned it off yet | 16:06 |
| anteaya | SergeyLukjanov: we have some patches we need merged | 16:06 |
| anteaya | for the upgrade | 16:06 |
| SergeyLukjanov | oh, got it | 16:07 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add vim to the list of explicit things on servers https://review.openstack.org/90786 | 16:07 |
| mordred | (that above is just to get it out of my local stack so I don't forget about it - super low priority) | 16:07 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: update gerrit project configs to use new Workflow label https://review.openstack.org/79107 | 16:07 |
| sweston | anteaya: hello! thanks for the links. for what it's worth, I just finished the upgrade script testing, with no problems. | 16:08 |
| *** darraghb has quit IRC | 16:08 | |
| *** afazekas has quit IRC | 16:08 | |
| *** adalbas has joined #openstack-infra | 16:08 | |
| anteaya | sweston: thank you | 16:08 |
| fungi | zaro: see if https://review.openstack.org/79107 looks right to you | 16:09 |
| anteaya | it is heartening to hear | 16:09 |
| zaro | fungi: looking | 16:09 |
| *** marcoemorais has quit IRC | 16:09 | |
| sweston | anteaya: ;-) always welcome | 16:09 |
| mordred | fungi: that patch looks good to me - do you want me to check every file? I've looked at a good pile of them | 16:10 |
| fungi | mordred: nah, as long as it looks like it didn't have obvious unintended side effects | 16:10 |
| mordred | looks consistent and I think I've seen an example of most of the forms it takes | 16:10 |
| jeblair | mordred: ditto | 16:10 |
| zaro | fungi: spot checked, LGTM | 16:11 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Add change abandoned logic to jeepyb https://review.openstack.org/86382 | 16:11 |
| *** marcoemorais has joined #openstack-infra | 16:11 | |
| *** darraghb has joined #openstack-infra | 16:11 | |
| openstackgerrit | A change was merged to openstack-infra/config: update gerrit core branch https://review.openstack.org/88038 | 16:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch review.o.o out of testmode https://review.openstack.org/90767 | 16:13 |
| fungi | i assume that one ^ was intended to be merged with the rest | 16:13 |
| *** UtahDave has joined #openstack-infra | 16:13 | |
| *** denis_makogon_ has joined #openstack-infra | 16:13 | |
| fungi | i can revert if we need it not applied until after we test things | 16:13 |
| *** pblaho has joined #openstack-infra | 16:14 | |
| jeblair | fungi: it should be merged now | 16:14 |
| *** denis_makogon_ is now known as denis_makogon | 16:15 | |
| zaro | i'm not sure whether this should be before or after upgrade can someone look? https://review.openstack.org/#/c/81622 | 16:16 |
| jeblair | zaro: before, otherwise we'll end up downgrading review | 16:16 |
| *** hashar has quit IRC | 16:17 | |
| zaro | ok, good thing i ask. | 16:17 |
| fungi | zaro: is the change number on line 39 of https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes correct? it doesn't seem to match the change subject, and was merged a while ago | 16:17 |
| *** annegentle has joined #openstack-infra | 16:17 | |
| jeblair | zaro: the puppet config for the future state needs to be finalized and merged before we can stop old review | 16:17 |
| *** packet has joined #openstack-infra | 16:18 | |
| zaro | fungi: 69800 was already merged. it was for review-dev.o.o | 16:19 |
| openstackgerrit | A change was merged to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 16:19 |
| zaro | fungi: the equivalent chane for review.o.o is 82416 | 16:19 |
| zaro | fungi: it's already on the list | 16:20 |
| fungi | zaro: yep, just noticed | 16:20 |
| fungi | thanks | 16:20 |
| Alex_Gaynor | I'm assuming it's a known thing that the py3k/pypy builds (which share the same buidlers) either aren't staritng, or have an incredibly deep queue? | 16:20 |
| anteaya | Alex_Gaynor: hi | 16:20 |
| anteaya | we are in the middle of upgrading gerrit right now, sorry about that | 16:21 |
| fungi | Alex_Gaynor: yeah, apparently we can't stop rackspace file injection of resolv.conf without also breaking their file injection of the interfaces config | 16:21 |
| fungi | Alex_Gaynor: should be resolved in a little while, hopefully | 16:21 |
| anteaya | Alex_Gaynor: what fungi said | 16:21 |
| zaro | i'm removing 69800 from the list of changes for upgrading. | 16:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Activate secondary index support for Gerrit https://review.openstack.org/79005 | 16:21 |
| Alex_Gaynor | fungi: oh, uh, good work us, I suppose | 16:21 |
| mordred | Alex_Gaynor: :) | 16:22 |
| jeblair | zaro: just mark it merged | 16:22 |
| zaro | jeblair: done | 16:22 |
| openstackgerrit | A change was merged to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o https://review.openstack.org/82416 | 16:22 |
| openstackgerrit | A change was merged to openstack-infra/config: update gerrit project configs to use new Workflow label https://review.openstack.org/79107 | 16:23 |
| fungi | okay, i think that's the last of them | 16:23 |
| zaro | fungi: don't forget 81622 | 16:23 |
| *** bauzas has joined #openstack-infra | 16:23 | |
| mordred | https://review.openstack.org/#/c/81622 | 16:23 |
| mordred | oh. what zaro said | 16:23 |
| fungi | ahh, someone moved it further up the file before i got to it | 16:24 |
| *** alexpilotti has quit IRC | 16:24 | |
| zaro | fungi: was me, sorry i didn't mention that. | 16:24 |
| openstackgerrit | A change was merged to openstack-infra/config: upgrade review.o.o to gerrit ver 2.8.4 https://review.openstack.org/81622 | 16:24 |
| *** jcoufal has quit IRC | 16:25 | |
| jeblair | anyone need a break before we shut it down? | 16:25 |
| *** aconrad has joined #openstack-infra | 16:25 | |
| mordred | I'm ready to do | 16:25 |
| mordred | I have coffee sitting next to me and just finished a bio break | 16:25 |
| * zaro is ready! | 16:25 | |
| fungi | i should probably force in those chattr changes which were approved before we shut down gerrit? | 16:26 |
| jeblair | fungi: oh yeah good idea | 16:26 |
| zaro | just double checked, looks like all changes have merged | 16:26 |
| fungi | since they haven't gotten to the gate due to the backlog they fix | 16:26 |
| *** jlibosva has quit IRC | 16:27 | |
| zaro | should i join OFTC now | 16:27 |
| *** msabramo has joined #openstack-infra | 16:27 | |
| mordred | jeblair: I just put in a note - one of the things in the manual list can't be done until gerrit is started on new gerrit | 16:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Overrwrite resolve.conf at boot time https://review.openstack.org/90763 | 16:27 |
| sweston | i tried joining OFTC, looks like I'm the only one there | 16:27 |
| anteaya | zaro: if you can easily | 16:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove chattr https://review.openstack.org/90764 | 16:28 |
| mordred | sweston: in #openstack-infra ? | 16:28 |
| jeblair | mordred: where's your note? | 16:28 |
| anteaya | sweston: oftc #openstack-infra ? | 16:28 |
| mordred | jeblair: I just added it to the other list - let me put it in the manual list too | 16:28 |
| anteaya | sweston: you should see me fungi jeblair mordred | 16:28 |
| anteaya | and a few others | 16:28 |
| jeblair | mordred: we have 3 etherpads, i don't know which ones you are talking abou | 16:28 |
| fungi | jeblair: okay, those are merged too now, so i can get some images rebuilding in rax when i get a breather | 16:28 |
| mordred | jeblair: good point | 16:28 |
| zaro | anteaya: i have connected, how do i join? | 16:28 |
| mordred | jeblair: I have put the node in all of the places I think it needs to be noted now | 16:29 |
| jeblair | fungi: why don't you go ahead and run puppet locally on nodepool and kick one off | 16:29 |
| anteaya | zaro /join #openstack-infra | 16:29 |
| fungi | jeblair: will do. if that seems to succeed, i should probably just delete all existing rackspace images since they're all broken | 16:29 |
| zaro | anteaya: i'm connected to both freenode and oftc, how do i let my client know that i want to join the oftc one and not the freenode one? | 16:30 |
| *** jpich has quit IRC | 16:30 | |
| jeblair | fungi: good point; actually you could just go ahead and delete them | 16:30 |
| *** dizquierdo has quit IRC | 16:30 | |
| anteaya | zaro: not sure, what client? | 16:30 |
| jeblair | mordred: got it | 16:30 |
| zaro | anteaya: weechat | 16:30 |
| mordred | zaro: I do it by switching to the oftc server tab and typing /join #openstack-infra in there | 16:30 |
| anteaya | zaro: go to the first window, click control-x | 16:30 |
| jeblair | mordred: zaro i think ctrl-x works in weechat and irssi | 16:31 |
| jeblair | what anteaya said | 16:31 |
| mordred | ++ | 16:31 |
| anteaya | you should see in the bar server weechat, server freenode, server oftc | 16:31 |
| anteaya | when you see server oftc /join #openstack-infra | 16:31 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Correct test_remote to support branchs without upstream https://review.openstack.org/83798 | 16:31 |
| *** eharney has quit IRC | 16:32 | |
| *** talluri has quit IRC | 16:32 | |
| jeblair | fungi, mordred: ready for shutdown? | 16:33 |
| zaro | anteaya: thanks. i'm on | 16:33 |
| anteaya | zaro: well done | 16:33 |
| fungi | jeblair: sure. are we already zuul queue dumped and killed? | 16:33 |
| mordred | jeblair: ++ | 16:33 |
| *** yfried has joined #openstack-infra | 16:33 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Correct test_remote to support branchs without upstream https://review.openstack.org/83798 | 16:33 |
| *** mrodden has quit IRC | 16:33 | |
| jeblair | fungi: no, will do after i shutdown; i'll do the next few tasks and let you know when we're ready to proceed | 16:34 |
| fungi | k | 16:34 |
| fungi | good by me then | 16:34 |
| mordred | jeblair: hrm. should we have an apache "down for maint" page? | 16:34 |
| jeblair | #status alert Gerrit is unavailable until further notice for a major upgrade. See: https://wiki.openstack.org/wiki/GerritUpgrade | 16:34 |
| openstackstatus | jeblair: sending alert | 16:34 |
| jeblair | mordred: maybe, but i don't think we're organized well enough for that this time | 16:35 |
| jeblair | mordred: i expect it's going to be up and down for a bit | 16:35 |
| mordred | jeblair: ++ | 16:36 |
| *** sahid has joined #openstack-infra | 16:36 | |
| -openstackstatus- NOTICE: Gerrit is unavailable until further notice for a major upgrade. See: https://wiki.openstack.org/wiki/GerritUpgrade | 16:36 | |
| *** ChanServ changes topic to "Gerrit is unavailable until further notice for a major upgrade. See: https://wiki.openstack.org/wiki/GerritUpgrade" | 16:36 | |
| *** thomasbiege has joined #openstack-infra | 16:36 | |
| *** dkliban_afk is now known as dkliban | 16:36 | |
| fungi | stopping puppet on nodepool and applying from a dev env... the make-resolv-conf-immutable notify is broken | 16:36 |
| jeblair | fungi: oops | 16:37 |
| mordred | fungi: piddle | 16:37 |
| *** thomasbiege has quit IRC | 16:37 | |
| jeblair | okay, gerrit is stopped and zuul is saved | 16:37 |
| mordred | I'm going to tranfer the database contents | 16:37 |
| jeblair | ack | 16:38 |
| jeblair | i assigned tasks in https://etherpad.openstack.org/p/J8WNqyP2dC | 16:38 |
| openstackstatus | jeblair: finished sending alert | 16:38 |
| bauzas | folks, I know you're in the middle of a major upgrade, but would it be worth pushing an haproxy reverseproxy in front of review.o.o so we could get a nice 404 page instead of a 503 error when looking now ? :) | 16:38 |
| *** aconrad has quit IRC | 16:38 | |
| jeblair | bauzas: you're welcome to contribute that next time | 16:39 |
| bauzas | I can propose a bp on that :) | 16:39 |
| bauzas | jeblair: coolio, will do | 16:39 |
| jeblair | bauzas: better to propose a change | 16:39 |
| bauzas | jeblair: good luck, folks, I'm with you ! | 16:39 |
| mordred | ERROR 2005 (HY000): Unknown MySQL server host '3eb5ccc86b48ecbd12e2e9c3cb5de78eac20f689.rackspaceclouddb.com' (0) | 16:40 |
| * mordred investigating | 16:41 | |
| jeblair | dns is updated; 5 minute ttl | 16:41 |
| mordred | oh. there is no review db | 16:42 |
| jeblair | i rather thought i created on | 16:42 |
| *** gokrokve has joined #openstack-infra | 16:44 | |
| mordred | there was a host and password entry in hiera - | 16:45 |
| *** msabramo has quit IRC | 16:45 | |
| fungi | did we recently change permissions on authorized_keys files? lots of puppet apply failures for all of them because they were 0400 so root thought it shouldn't write to them | 16:45 |
| *** ihrachyshka has quit IRC | 16:45 | |
| mordred | we changed puppet to use the ssh_authorized_key method | 16:45 |
| *** mrodden has joined #openstack-infra | 16:46 | |
| fungi | strange. it seemed to not like it at any rate | 16:46 |
| mordred | new cloud db host in hiera | 16:46 |
| fungi | i don't suppose we ever heard back on why trove instances are reachable on 3306/tcp from other tenants | 16:48 |
| jeblair | This is known and intentional. Just like Cloud Servers, Cloud DBs have an open connection to the datacenter which means all other devices in the datacenter. They don't have public IP addresses and as far as I know they aren't ever going to support Cloud Networks. If you need more control over the connection (or the my.cnf file) it would be much better to deploy MySQL on a Cloud Server. | 16:48 |
| * fungi sighs loudly | 16:49 | |
| mordred | wow | 16:49 |
| jeblair | fungi: would you like to move back to server-local mysql? | 16:49 |
| *** gokrokve has quit IRC | 16:49 | |
| mordred | fwiw, the trove team IS working on neutron support for this cycle | 16:49 |
| jeblair | it should be a fairly small config change to switch back | 16:50 |
| fungi | jeblair: it's worth strong consideration, but for now maybe we just hope there are no pre-authentication vulnerabilities for the mysql version in use at rax, keep our instance names hidden away in hiera and hope for the best :/ | 16:51 |
| fungi | or do you mean for this maintenance specifically | 16:51 |
| jeblair | fungi: either. :) | 16:51 |
| *** sweston is now known as sweston_ | 16:51 | |
| jeblair | fungi: sounds like proceed as planned for this maint, and then we can anguish over whether to move back later | 16:51 |
| jeblair | (it should be a trivial downtime to move back) | 16:52 |
| fungi | yeah, i don't want to unnecessarily extend the down-time for now, and it merits further discussion | 16:52 |
| mordred | vipul: ^^ just fyi - the above is a stupid conversation for us to need to have | 16:52 |
| mordred | SlickNik: ^^ (I'd ping hub_cap too - but he doesn't seem to be on) | 16:53 |
| jeblair | fungi: are you available for upgrade work or are you still on nodeppol? | 16:53 |
| mordred | mysql> select count(*) from changes \G | 16:53 |
| mordred | *************************** 1. row *************************** | 16:53 |
| mordred | count(*): 90287 | 16:53 |
| mordred | database transferred | 16:53 |
| fungi | jeblair: i'm futzing with nodepool... my latest image-update just met with 503 SERVICE UNAVAILABLE from bitbucket.org | 16:54 |
| *** derekh has quit IRC | 16:54 | |
| jeblair | fungi: gah ok, i'll take the thing i assigned to you for the upgrade | 16:54 |
| fungi | but i should be able to work on upgradey things now | 16:54 |
| fungi | looking | 16:54 |
| fungi | sorry | 16:54 |
| mordred | fungi: btw - I've got a patch to remove the need for that bitbucket stuff | 16:54 |
| jeblair | fungi: np | 16:54 |
| yfried | Hi guys, I'm trying to setup logstash on my jenkins, according to how the gate is set, http://paste.openstack.org/show/77483/ and it keeps failing http://paste.openstack.org/show/77478/ | 16:54 |
| anteaya | yfried: hi | 16:54 |
| jeblair | fungi: i'll give it back to you :) | 16:54 |
| yfried | anyone familiar with that? | 16:55 |
| anteaya | right now we are in the middle of a gerrit upgrade | 16:55 |
| fungi | jeblair: working on it now | 16:55 |
| anteaya | yfried: after the upgrade we can talk about your logstash | 16:55 |
| fungi | also, retrying the image update confirms bitbucket really seems to be toes-up | 16:55 |
| jeblair | fungi: see etherpad -- i looked up the hiera keys for you | 16:56 |
| jeblair | fungi: (we never got around to doing this independently for heartbleed, so we rolled it into this upgrade) | 16:56 |
| *** sarob has joined #openstack-infra | 16:56 | |
| anteaya | fungi: yes bitbucket confirms performance issues: http://status.bitbucket.org/ | 16:56 |
| *** hogepodge has joined #openstack-infra | 16:56 | |
| *** sdake_ has quit IRC | 16:57 | |
| jeblair | mordred: are you interested in running the upgrade script? | 16:57 |
| mordred | jeblair: sure. | 16:57 |
| jeblair | mordred: all yours then; let me know if you need anything | 16:58 |
| mordred | well, the script isn't goig to work as written... | 16:58 |
| mordred | let me edit it real quick | 16:59 |
| mordred | (it's assuming local db) | 16:59 |
| *** harlowja_away is now known as harlowja | 17:00 | |
| zaro | mordred: could you check to see if there's an 'account_group_by_id' table in the db? | 17:00 |
| jeblair | zaro, mordred: i'll do that | 17:01 |
| sdague | fungi: bitbucket definitely had a big outage this weekend, it would not surprise me if they were still recovering | 17:01 |
| *** SumitNaiksatam has quit IRC | 17:01 | |
| jeblair | ERROR 1146 (42S02): Table 'reviewdb.account_group_by_id' doesn't exist | 17:02 |
| zaro | mordred, jeblair : ohh nice, then that TRUNCATE TABLE workaround is _not_ needed. | 17:03 |
| fungi | jeblair: new host keypairs (rsa and dsa) committed in hiera now | 17:03 |
| jeblair | fungi: cool, thanks | 17:03 |
| jeblair | fungi: btw, when we're done, would you mind sending a gpg-signed msg to openstack-dev with the host keys in it? | 17:03 |
| jeblair | er, the fingerprints | 17:03 |
| jeblair | fungi: the docs we published mention the web verification process, but some folks asked for a gpg signed mail too | 17:04 |
| fungi | jeblair: gladly! i get it drafted up in a few minutes so it's ready to fire when the time comes | 17:04 |
| fungi | er, i'll | 17:04 |
| *** melwitt has joined #openstack-infra | 17:04 | |
| *** moted has quit IRC | 17:04 | |
| jeblair | fungi: i figured you probably have the larger web of trust in the community | 17:04 |
| zaro | mordred: i just updated the GERRIT_WAR & GERRIT_DOWNLOAD_URL vars | 17:04 |
| boris-42 | https://pbs.twimg.com/media/BmUr9B2CcAAi_BH.png | 17:04 |
| jeblair | boris-42: :) | 17:05 |
| davidlenwell | boris-42: lol | 17:05 |
| *** bauzas has quit IRC | 17:05 | |
| *** talluri has joined #openstack-infra | 17:05 | |
| *** BobBall is now known as BobBallAway | 17:06 | |
| *** lakshmi has joined #openstack-infra | 17:06 | |
| *** Ryan_Lane has joined #openstack-infra | 17:08 | |
| mordred | jeblair: it says "GERRIT_ADMIN=corvus" | 17:08 |
| zaro | mordred: yeah, that's for review-dev. you'll have to change that | 17:08 |
| jeblair | oO | 17:09 |
| yfried | anteaya: what do you mean "after upgrade"? | 17:09 |
| mordred | I think it will be better to run the aggreement bit from the end of the script by hand as a second step | 17:09 |
| anteaya | yfried: I mean that any conversation about anything other than the upgrade is a distraction right now | 17:10 |
| zaro | mordred: i made a few last minute edits to the script. just updated the JAVAMELOODY_PLUGIN var as well. | 17:10 |
| mordred | yeah - I'll grab those in a second | 17:10 |
| anteaya | yfried: so after the upgrade is finished we can talk about other issues, including your logstash questions | 17:10 |
| *** esker has joined #openstack-infra | 17:10 | |
| anteaya | yfried: tomorrow might be a good time | 17:10 |
| mordred | zaro: why does gerrit need to be started if the all-projects git interaction is with the copy on the filesystem? | 17:11 |
| zaro | which line? | 17:11 |
| mordred | git clone $GERRIT_SITE/git/All-Projects.git | 17:11 |
| *** sweston_ has quit IRC | 17:12 | |
| yfried | anteaya: tnx. didn't know about it. sorry. last question can you point me to anyone I can bother privatly who might know? | 17:12 |
| zaro | mordred: i assumed you answered your own question there? | 17:13 |
| anteaya | yfried: np, well clarkb is our best logstash resource, he would be helping with the upgrade if he were here but he is in hawaii this week | 17:13 |
| mordred | yeah | 17:13 |
| anteaya | yfried: so try us tomorrow, we will do the best we can | 17:13 |
| mordred | jeblair: would you please double-check /root/upgrade_script.sh on new review for me? | 17:15 |
| mordred | jeblair: I've made a few edits and changes and I think they're good but would like second eyes | 17:15 |
| zaro | yfried: you might want to ping jesusaurus | 17:15 |
| jeblair | mordred: ack | 17:15 |
| *** msabramo has joined #openstack-infra | 17:16 | |
| *** nati_ueno has joined #openstack-infra | 17:16 | |
| *** markmc has quit IRC | 17:16 | |
| yfried | anteaya: tnx | 17:16 |
| jeblair | JAVAMELODY_PLUGIN=javamelody-$git_sha.jar | 17:16 |
| yfried | zaro: tnx | 17:16 |
| jeblair | mordred, zaro: ^ ? | 17:16 |
| zaro | updated on etherpad | 17:16 |
| mordred | jeblair: thanks- javamelody-4744bfb.jar | 17:16 |
| jesusaurus | yfried: you have logstash questions? | 17:16 |
| mordred | once gerrit is running, I'll copy accepted_agreement_users.txt back to my machine and run it as me | 17:17 |
| jeblair | jesusaurus, yfried: would you mind taking the conversation to another channel? perhaps #openstack-qa or #openstack-dev | 17:17 |
| jeblair | mordred: will you update upgrade_script.sh? | 17:17 |
| mordred | jeblair: yes | 17:17 |
| jeblair | with the javamelody change | 17:17 |
| *** sdake_ has joined #openstack-infra | 17:18 | |
| *** sdake_ has quit IRC | 17:18 | |
| *** sdake_ has joined #openstack-infra | 17:18 | |
| mordred | done | 17:18 |
| jeblair | mordred: cp -rp $GERRIT_SITE $GERRIT_SITE-backup | 17:18 |
| jeblair | mordred: suggest removing that line | 17:18 |
| mordred | jeblair: nod | 17:18 |
| jeblair | mordred: actuall rollback strategy is use old host | 17:18 |
| mordred | done | 17:18 |
| *** zzelle has joined #openstack-infra | 17:19 | |
| *** zhiyan is now known as zhiyan_ | 17:19 | |
| jeblair | mordred: lgtm; if i were you i'd actually just run that one line at a time in an interactive shell | 17:20 |
| *** ominakov has quit IRC | 17:20 | |
| mordred | jeblair: ++ | 17:20 |
| mordred | jeblair: also, I just made one more change | 17:20 |
| *** SumitNaiksatam has joined #openstack-infra | 17:21 | |
| mordred | I made a /root/.my.cnf with the db stuff | 17:21 |
| mordred | so the script should now be ehterpad-able | 17:21 |
| jeblair | mordred: cool | 17:21 |
| jeblair | mordred: fire at will | 17:22 |
| mordred | ok. I've put it into the etherpad | 17:22 |
| mordred | rocking and rolling | 17:22 |
| mordred | zaro: root@review:/home/gerrit2# /usr/bin/java -jar gerrit-wars/$GERRIT_WAR -d $GERRIT_SITE --batch --no-auto-start | 17:24 |
| mordred | fatal: unknown command -d | 17:24 |
| *** gokrokve has joined #openstack-infra | 17:24 | |
| *** praneshp has joined #openstack-infra | 17:24 | |
| *** atiwari_ has quit IRC | 17:24 | |
| zaro | what? there's definately a '-d' option | 17:24 |
| anteaya | zaro: is there a missing library? | 17:25 |
| anteaya | or dependency? | 17:25 |
| *** yassine has quit IRC | 17:25 | |
| mordred | this expands to /usr/bin/java -jar gerrit-wars/gerrit-v2.8.4.15.6dc8444.war -d /home/gerrit2/review_site --batch --no-auto-start | 17:25 |
| mordred | and the war is there | 17:25 |
| zaro | mordred: can you run with --help? | 17:26 |
| *** ArxCruz has quit IRC | 17:26 | |
| mordred | it's missing the word init | 17:26 |
| jeblair | mordred: ++ just came to that conclusion | 17:26 |
| *** chandan_kumar has joined #openstack-infra | 17:26 | |
| *** sdake_ has quit IRC | 17:28 | |
| *** SumitNaiksatam has quit IRC | 17:28 | |
| *** gokrokve has quit IRC | 17:29 | |
| *** aconrad has joined #openstack-infra | 17:29 | |
| *** jp_at_hp has quit IRC | 17:31 | |
| *** ArxCruz has joined #openstack-infra | 17:31 | |
| mordred | it's doing the migration of data, btw | 17:32 |
| *** amcrn has joined #openstack-infra | 17:32 | |
| *** yidclare has joined #openstack-infra | 17:32 | |
| mordred | it's currently migrating to ... ah, it's one | 17:32 |
| mordred | done | 17:32 |
| *** thedodd has joined #openstack-infra | 17:32 | |
| *** thomasbiege has joined #openstack-infra | 17:32 | |
| mordred | it printed a set of sql it thinks I should run to delete old things | 17:32 |
| zaro | yay!!! | 17:32 |
| mordred | shoudl I do that? | 17:32 |
| anteaya | can you paste? | 17:32 |
| mordred | I have added to teh bottom of the upgrade script etherpad | 17:33 |
| mordred | jeblair: ^^ | 17:33 |
| mordred | I'm going ot not do it now because it does not seem urgent | 17:33 |
| *** ihrachyshka has joined #openstack-infra | 17:33 | |
| zaro | ++ | 17:33 |
| jeblair | mordred: why don't i go ahead and do that while you continue? | 17:34 |
| mordred | ++ | 17:34 |
| mordred | zaro: there is now "javamelody-4744bfb.jar" in review_site/plugins - shoudl that be javamelody.jar ? | 17:34 |
| *** prad_ has quit IRC | 17:34 | |
| zaro | i don't tihnk filename matters. | 17:35 |
| zaro | mordred: i would just leave as is | 17:35 |
| jeblair | sql cleanup done | 17:35 |
| mordred | root@review:/home/gerrit2# mysql reviewdb -e "INSERT INTO account_group_names(group_id,name) VALUES('$ICLA_GROUP_ID','CLA Accepted - ICLA')" | 17:37 |
| mordred | ERROR 1062 (23000) at line 1: Duplicate entry 'CLA Accepted - ICLA' for key 'PRIMARY' | 17:37 |
| *** talluri has quit IRC | 17:38 | |
| fungi | mordred: i believe gerrit auto-creates those groups on upgrade | 17:38 |
| jeblair | | 324 | CLA Accepted - ICLA | | 17:39 |
| jeblair | | 325 | Change Owner | | 17:39 |
| jeblair | last two groups | 17:39 |
| jeblair | so they were recently created | 17:39 |
| fungi | since it switches all clas to be group-clas beginning with 2.5.x (when they added a mechanism to auto-accept by automatically adding users to the corresponding group) | 17:39 |
| mordred | awesome | 17:40 |
| mordred | in taht case, I believe the upgrade script is done | 17:40 |
| jeblair | https://code.google.com/p/gerrit/issues/detail?id=2377 | 17:40 |
| mordred | except for pushing in the cla signers | 17:40 |
| *** eharney has joined #openstack-infra | 17:40 | |
| jeblair | according to the bug, zaro decided it wasn't necessary after all | 17:40 |
| zaro | yeah that was an invalid bug | 17:41 |
| mordred | for the record, I've been updating the upgrade_script.sh in root - so it reflects what was run | 17:41 |
| *** sweston has joined #openstack-infra | 17:41 | |
| fungi | well, adding the accounts to that group is needed. it wasn't a bug upstream because their non-group cla implementation in 2.4.x was half-baked and thus unsupportable without patches we added to fix it | 17:41 |
| jeblair | mordred: so are we at line 144 in https://etherpad.openstack.org/p/gerrit_upgrade_script | 17:42 |
| jeblair | fungi: yeah, i think that's next | 17:42 |
| *** e0ne has joined #openstack-infra | 17:42 | |
| mordred | yes | 17:42 |
| mordred | jeblair: next step is "start gerrit" | 17:42 |
| fungi | so they never added a mechanism to upgrade from their broken implementation | 17:42 |
| anteaya | for those following along, this is our upgrade todo list we are working though: https://etherpad.openstack.org/p/J8WNqyP2dC | 17:43 |
| jeblair | mordred: pause for a sec | 17:43 |
| jeblair | mordred: it's not clear to me whether we should run puppet now, then start gerrit... or the other way around | 17:43 |
| jeblair | mordred: i think we should run puppet now, then start gerrit | 17:44 |
| sweston | in my experience, if puppet is working correctly, it is better to let puppet do it | 17:44 |
| jeblair | then proceed with the upgrade script | 17:44 |
| mordred | jeblair: kk | 17:44 |
| mordred | jeblair: I agree | 17:44 |
| mordred | jeblair: would you like to do that? | 17:44 |
| *** hemna has joined #openstack-infra | 17:44 | |
| jeblair | mordred: i will do that | 17:44 |
| *** thomasbiege has quit IRC | 17:44 | |
| jeblair | fungi: how's the resolv.conf fix coming? :) | 17:45 |
| mordred | fungi: for the bitbucket ... | 17:45 |
| fungi | jeblair: http://paste.openstack.org/show/77495/ | 17:45 |
| mordred | fungi: find any occurance of use of or download of ez_setup.py and remove it | 17:45 |
| jeblair | i'm not asking in relation to nodepool; i'm asking because we need that fix for all servers | 17:46 |
| fungi | jeblair: i'm all set to push that for review when there's a gerrit | 17:46 |
| *** prad_ has joined #openstack-infra | 17:46 | |
| jeblair | fungi: we're getting rid of managed resolv.conf? | 17:46 |
| fungi | jeblair: an already merged change sets it via rc.local | 17:47 |
| jeblair | fungi: that was for nodepool | 17:47 |
| jeblair | mordred: right? | 17:47 |
| fungi | oh, then we should just remove the notify => Exec['make-resolv-conf-immutable'], line | 17:47 |
| jeblair | fungi: yeah i think so | 17:47 |
| fungi | updating my prepared patch in that case | 17:47 |
| mordred | ++ | 17:48 |
| *** morganfainberg_Z is now known as morganfainberg | 17:48 | |
| jeblair | i'm updating my environment on puppetmaster with that change | 17:48 |
| jeblair | puppet agent --test --puppetdlockfile=/tmp/alt-lock-file --environment=corvus | 17:49 |
| jeblair | running that ^ | 17:49 |
| mordred | ++ | 17:49 |
| *** arnaud__ has joined #openstack-infra | 17:50 | |
| fungi | not as easy to test for nodepool image updates since they want to pull the config repo themselves during build | 17:51 |
| fungi | however bitbucket stopped failing at least | 17:51 |
| *** darraghb has quit IRC | 17:51 | |
| anteaya | fungi: because it is up or because you removed ez_setup.py? | 17:51 |
| jeblair | mordred: okay, puppet has been run; i think you can start gerrit now | 17:51 |
| mordred | jeblair: starting gerit | 17:52 |
| fungi | anteaya: bitbucket seems to be returning the file again | 17:52 |
| mordred | Starting Gerrit Code Review: FAILED | 17:52 |
| anteaya | fungi: k | 17:52 |
| jeblair | nothing in the error log | 17:53 |
| *** markwash has joined #openstack-infra | 17:53 | |
| mordred | I'm going to run start.sh by hand | 17:53 |
| mordred | or, rather | 17:53 |
| mordred | yup. tried again. still errror. still nothing in logs | 17:54 |
| *** mrmartin has joined #openstack-infra | 17:54 | |
| fungi | mordred: one possibility... if you updated the all-projects acl config file but not the groups file (which lists the corresponding uuids from the db for groups mentioned in the acl) then it might be unhappy? | 17:54 |
| anteaya | is there a shell debugger you can run it in? | 17:54 |
| mordred | fungi: I did not update the all projects acls yet | 17:55 |
| sweston | what about starting manually ? java -jar ... gerrit.war daemon -d ... review_site | 17:55 |
| fungi | mordred: okay, thinking harder in that case ;) | 17:55 |
| mordred | there is no gerrit.war | 17:55 |
| *** hogepodge has quit IRC | 17:56 | |
| jeblair | mordred: there isn't? | 17:56 |
| zaro | ohh i see the problem. | 17:56 |
| anteaya | isnt this the war? gerrit-wars/gerrit-v2.8.4.15.6dc8444.war | 17:56 |
| zaro | download location is incorrect | 17:56 |
| *** hogepodge has joined #openstack-infra | 17:56 | |
| mordred | wait - I'm wrong | 17:56 |
| *** mat-lowery has joined #openstack-infra | 17:56 | |
| jeblair | -rw-r--r-- 1 gerrit2 gerrit2 35115174 Apr 28 17:56 /home/gerrit2/review_site/bin/gerrit.war | 17:56 |
| sweston | there should be a gerrit war file in the gerrit2 home directory .. /home/gerrit2/gerrit-wars/gerrit-version.war | 17:56 |
| jeblair | if you start it manually, be sure to do so as the gerrit2 user | 17:57 |
| mordred | yup. trying that now | 17:57 |
| *** e0ne has quit IRC | 17:57 | |
| mordred | it is silent | 17:57 |
| mordred | is there a way to start it in non-daemon mode? | 17:57 |
| *** nati_uen_ has joined #openstack-infra | 17:58 | |
| jeblair | mordred: ~gerrit2 is owned by root | 17:58 |
| jeblair | mordred: i'm not sure if that's an issue but let's fix that real quick and retry | 17:58 |
| mordred | so it is | 17:58 |
| *** ramashri has joined #openstack-infra | 17:59 | |
| jeblair | done | 17:59 |
| *** aysyd has quit IRC | 17:59 | |
| mordred | still failed | 17:59 |
| *** nati_uen_ has quit IRC | 17:59 | |
| mordred | but now we have error logs! | 17:59 |
| mordred | the db settings are wrong | 17:59 |
| mordred | in gerrit.config | 17:59 |
| mordred | I believe we may have missed a puppet patch | 18:00 |
| *** sandywalsh has quit IRC | 18:00 | |
| mordred | I can fix manually | 18:00 |
| *** marcoemorais has quit IRC | 18:00 | |
| jeblair | mordred: before you fix manually | 18:00 |
| *** sandywalsh has joined #openstack-infra | 18:00 | |
| *** e0ne has joined #openstack-infra | 18:00 | |
| jeblair | mordred: what puppet patch do we need? | 18:00 |
| *** afazekas has joined #openstack-infra | 18:00 | |
| mordred | we need something that sets the host to not be localhost | 18:00 |
| *** packet has quit IRC | 18:00 | |
| mordred | the other settings are all correct | 18:00 |
| jeblair | ah ok | 18:00 |
| jeblair | mordred: let me see if i can fix that in puppet real quick | 18:00 |
| mordred | kk | 18:01 |
| sweston | you do not have anything in the /home/gerrit2/review_site/logs directory? | 18:01 |
| *** packet has joined #openstack-infra | 18:01 | |
| mordred | sweston: we do now | 18:01 |
| mordred | jeblair: the correct host is in hiera, btw | 18:01 |
| *** hashar has joined #openstack-infra | 18:01 | |
| *** nati_ueno has quit IRC | 18:01 | |
| hashar | have a good Gerrit upgrade folks! | 18:01 |
| sweston | mordred: ok, that's good at least we have an error to work off of | 18:01 |
| mordred | hashar: we're getting closer | 18:01 |
| mordred | sweston: it's a mysql config error - we're fixing | 18:02 |
| *** marcoemorais has joined #openstack-infra | 18:02 | |
| sweston | mordred: ok, gotcha | 18:02 |
| *** nati_ueno has joined #openstack-infra | 18:03 | |
| *** dkliban is now known as dkliban_afk | 18:03 | |
| jeblair | so... i changed it in my environment but puppet still didn't touch the file | 18:04 |
| mordred | hrm | 18:05 |
| *** sarob has quit IRC | 18:06 | |
| mordred | jeblair: in modules/openstack_project/manifests/gerrit.pp | 18:06 |
| mordred | jeblair: mysql_host is not plumbed through to ::gerrit | 18:06 |
| jeblair | ah | 18:06 |
| jeblair | er how did that work on review_dev? | 18:07 |
| jeblair | perhaps it did not | 18:07 |
| *** bhuvan has joined #openstack-infra | 18:07 | |
| jeblair | mordred: okay, fixed | 18:08 |
| jeblair | mordred: clear to proceed | 18:08 |
| *** arnaud__ has quit IRC | 18:08 | |
| mordred | gerrit starting | 18:08 |
| mordred | failed | 18:08 |
| mordred | but tok much longer this time | 18:09 |
| *** sdake_ has joined #openstack-infra | 18:09 | |
| jeblair | com.google.inject.ProvisionException: Guice provision errors: | 18:09 |
| jeblair | 1) No index versions ready; run Reindex | 18:09 |
| mordred | oh - that's the secondary index stuff, right? | 18:09 |
| zaro | looks like it | 18:09 |
| mordred | on it | 18:10 |
| mordred | 20% done | 18:12 |
| *** pblaho has quit IRC | 18:13 | |
| *** thedodd has quit IRC | 18:13 | |
| mordred | 59 | 18:13 |
| mordred | gah | 18:13 |
| mordred | 50 | 18:13 |
| *** nati_ueno has quit IRC | 18:13 | |
| *** gokrokve has joined #openstack-infra | 18:14 | |
| anteaya | mordred is wearing the progress bar hat | 18:15 |
| *** zz_gondoi is now known as gondoi | 18:15 | |
| *** thomasbiege has joined #openstack-infra | 18:15 | |
| *** aysyd has joined #openstack-infra | 18:16 | |
| mordred | 96 | 18:16 |
| mordred | ok. it's up and running | 18:16 |
| *** zehicle_at_dell has joined #openstack-infra | 18:17 | |
| zaro | i'm in. | 18:17 |
| anteaya | so indexing is finished or gerrit is up? | 18:17 |
| mordred | I'm going to run the CLA setting script | 18:17 |
| mordred | gerrit is u | 18:17 |
| mordred | up | 18:17 |
| anteaya | cool | 18:17 |
| anteaya | well done | 18:17 |
| mordred | jeblair: any issues with me moving forward on steps on the manual list? | 18:17 |
| jeblair | anteaya: we're far from finished | 18:17 |
| * anteaya nods | 18:17 | |
| jeblair | mordred: go right ahead | 18:17 |
| mordred | I have three - run clas script, update acls, update dashboards | 18:17 |
| jeblair | mordred: do you think i can run manage-projects while you do that, or should i wait? | 18:18 |
| jeblair | (manage projects needs to run to update all the per-project acls) | 18:18 |
| mordred | jeblair: I see no reason why it can't run | 18:19 |
| jeblair | i will do that then | 18:19 |
| *** gokrokve has quit IRC | 18:19 | |
| hemna | gerrit good to submit reviews ? | 18:19 |
| *** _nadya_ has joined #openstack-infra | 18:20 | |
| anteaya | hemna: not yet | 18:20 |
| hemna | ok :) thanks | 18:20 |
| anteaya | hemna: we just achieved one of the steps on teh list | 18:20 |
| jeblair | manage-projects is using the local mysql db | 18:20 |
| hemna | ok np. I just happened to hit refresh in my browser and saw it was "up" | 18:20 |
| anteaya | yes, so the app is running now we need to fine tune settings | 18:21 |
| jeblair | fungi: do you know how manage-projects gets its database config? | 18:22 |
| fungi | jeblair: hmm... good question--looking now | 18:22 |
| *** thedodd has joined #openstack-infra | 18:23 | |
| jeblair | fungi, mordred: i think jeepyb is hardcoded for a local database | 18:23 |
| jeblair | as it, it simply does not use the hostname argument | 18:23 |
| mordred | jeblair: spectacular | 18:24 |
| jeblair | i will start hacking on that | 18:24 |
| fungi | http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/gerritdb.py#n54 | 18:24 |
| fungi | yep, seems to not specify a hostname, but could conceivably scrape that from the config too | 18:24 |
| *** yfried has quit IRC | 18:26 | |
| *** doddstack has joined #openstack-infra | 18:26 | |
| *** arnaud__ has joined #openstack-infra | 18:28 | |
| *** thedodd has quit IRC | 18:28 | |
| jeblair | okay, i have that change staged and ready to go; the original run is still going though and i'm a little afraid to interrupt it | 18:28 |
| jeblair | it has about 140 more projects to go | 18:28 |
| mordred | acl updates updated | 18:29 |
| *** nati_ueno has joined #openstack-infra | 18:29 | |
| mordred | important changes dashboard uploaded | 18:29 |
| *** marcoemorais has quit IRC | 18:29 | |
| jeblair | zaro: can you help mordred change the "CR" label to "R" ? | 18:30 |
| zaro | jeblair: can't. there's a bug in gerrit that prevents it. | 18:30 |
| *** _nadya_ has quit IRC | 18:30 | |
| jeblair | oh. that's unfortunate. | 18:31 |
| zaro | https://code.google.com/p/gerrit/issues/detail?id=2618 | 18:31 |
| *** marcoemorais has joined #openstack-infra | 18:31 | |
| mordred | I am not seeing an Approved state in workflow for things I should have approve on - oh, that's because we need per-project acls run in isn't it? | 18:31 |
| jeblair | mordred: yep. | 18:31 |
| mordred | I DO see work in progress where appropriate | 18:31 |
| mordred | how do I see dashboards? | 18:32 |
| anteaya | mordred: Projects -> List -> All Projects -> Dashboards | 18:32 |
| *** yfried has joined #openstack-infra | 18:33 | |
| mordred | great. important changes works | 18:33 |
| anteaya | yay | 18:33 |
| jeblair | zaro: are you sure order doesn't matter on the review labels? | 18:34 |
| zaro | jeblair: it's alpha order | 18:34 |
| jeblair | i could have sworn that when gerrit started, the order was V/CR/W | 18:34 |
| jeblair | then mordred made a change that removed an extra Verified label from the config file (which was before code-review) | 18:35 |
| jeblair | and then the order changed | 18:35 |
| morganfainberg | jeblair, i've see that setting not honored in newer versions of gerrit... it just does whatever it wants | 18:35 |
| zaro | jeblair: might have been on old important changes dashboard only | 18:35 |
| jeblair | no, was on the main screen | 18:35 |
| morganfainberg | jeblair, it seems to make it alphabetical? | 18:35 |
| jeblair | morganfainberg: ya, that's what zaro just said. i'm attempting to dispute that. :) | 18:35 |
| zaro | mordred: steps to create dashboards is in the gerrit upgrade script | 18:36 |
| jeblair | sadly, i lost my evidence in a reload | 18:36 |
| morganfainberg | jeblair, no i've seen exactly what you saw. | 18:36 |
| mordred | jeblair, zaro: I have tested just now by moving the code review block in the all projects file - and now CR is second | 18:36 |
| morganfainberg | jeblair, it was "correct" then switched to alpha after gerrit was running? logged in? something | 18:36 |
| zaro | jeblair: i've already verified that it's always alpha order. | 18:36 |
| *** ildikov has joined #openstack-infra | 18:36 | |
| mordred | zaro: I have just verified the opposite :) | 18:37 |
| jeblair | morganfainberg: oh, we agree then -- the change wasn't logging in -- it was actually mordred making a config change | 18:37 |
| morganfainberg | jeblair, hm. odd. i'd love to know what setting that was. | 18:37 |
| jeblair | morganfainberg: and he just changed it again so it should be back to the order we are accustomed to | 18:37 |
| jeblair | morganfainberg: it's simply the order listed in the config file | 18:37 |
| mordred | morganfainberg: it's the order of the sections in projects.config | 18:37 |
| morganfainberg | jeblair, oh good it's honoring that setting then | 18:37 |
| morganfainberg | jeblair, in .. 2.6 i think it just went alpha... or changed | 18:37 |
| zaro | mordred: what? i'm must be bonkers then. | 18:37 |
| * mordred hands zaro a wet cat | 18:37 | |
| morganfainberg | wait, mordred , you changed it back? | 18:38 |
| jeblair | mordred: i see cr/v/w now | 18:38 |
| morganfainberg | i still see it in alphabetical | 18:38 |
| *** markmcclain has joined #openstack-infra | 18:38 | |
| mordred | hrm. I see v/cr/w | 18:38 |
| morganfainberg | mordred, cr/v/w here. | 18:38 |
| mordred | have you logged out and logged back in? | 18:38 |
| anteaya | everybody refresh, no config changes | 18:38 |
| fungi | yep, lgtm now | 18:38 |
| morganfainberg | mordred, yep, just did | 18:38 |
| zaro | ohhh now i remember. i was testing on master, not 2.8 | 18:38 |
| mordred | WEIRD | 18:38 |
| jeblair | (i haven't logged in at all) | 18:39 |
| *** nati_ueno has quit IRC | 18:39 | |
| mordred | why are we seeing different things? | 18:39 |
| mordred | morganfainberg: where are you seeing yours? | 18:39 |
| morganfainberg | let me try incognito | 18:39 |
| mordred | AH | 18:39 |
| morganfainberg | mordred, https://review.openstack.org/#/q/status:open,n,z | 18:39 |
| mordred | wait | 18:39 |
| morganfainberg | logged in | 18:39 |
| mordred | we're talking about different things | 18:39 |
| mordred | you're talking about the status grid | 18:39 |
| jeblair | yep | 18:39 |
| morganfainberg | jeblair, yep | 18:39 |
| mordred | I'm taling about the order in which the review vote blocks are displayed | 18:39 |
| mordred | because, of course, those shoudl sort differently | 18:40 |
| morganfainberg | mordred, yep* | 18:40 |
| morganfainberg | mordred, ah yeah | 18:40 |
| jeblair | oh good grief | 18:40 |
| mordred | the vote block order is based on order of block in config file | 18:40 |
| morganfainberg | we might want to make those match... | 18:40 |
| zaro | i'm seeing cr/v/w | 18:40 |
| mordred | the matrix order is alpha | 18:40 |
| mordred | you'd think with all of those class factory factory class injection class factories there would be, you know, a sort method | 18:40 |
| morganfainberg | mordred, wait so i should see V/CR/W on a specific review? | 18:40 |
| morganfainberg | mordred, because... i see it alphabetical there too | 18:41 |
| mordred | if you click review | 18:41 |
| zaro | what do we want to change order on? i thought it was the columns on the dashboards? | 18:41 |
| mordred | zaro: it is - I was talking about something different, because I like to make things complex | 18:41 |
| *** dkliban_afk is now known as dkliban | 18:42 | |
| morganfainberg | mordred, ok so where should i see it V/CR/W? in the place where you sleect the score? | 18:42 |
| zaro | ok, good to know i'm not bonkers after all. | 18:42 |
| morganfainberg | zaro, nah we're all just talking about many different things :) | 18:42 |
| morganfainberg | zaro, hehe :) | 18:43 |
| *** alexpilotti has joined #openstack-infra | 18:43 | |
| mordred | morganfainberg: you probably cannot see it at all- when I'm logged in as a project bootstrapper, it shows me V/CR/W as the order of the vote blocks | 18:43 |
| zaro | yeah, columns are always alpha order. everybody agree? | 18:43 |
| mordred | zaro: yes | 18:43 |
| morganfainberg | mordred ah ok | 18:43 |
| jaypipes | what's this new "Workflow" column about? :) | 18:43 |
| morganfainberg | zaro, ++ yeah | 18:43 |
| fungi | yeah, when looking at the main view for a change i see the vote table with columns in alpha order | 18:43 |
| mordred | morganfainberg: and I verified that changing the block order in the config file DOES affect it | 18:43 |
| jeblair | all right, i don't think the order particularly matters for the review box, but let's set it to what we want it to be anyway: v/r/w | 18:43 |
| sweston | zaro: ++ yes, verified over here as well | 18:44 |
| morganfainberg | mordred, cool. yeah that makes sense. | 18:44 |
| jeblair | (it doesn't matter because no one sees verified) | 18:44 |
| mordred | jaypipes: it's where Approved or Work in Progress go now | 18:44 |
| jeblair | jaypipes: https://wiki.openstack.org/wiki/GerritUpgrade | 18:44 |
| mordred | jeblair: ++ | 18:44 |
| anteaya | jaypipes: https://wiki.openstack.org/wiki/GerritUpgrade#DNS_Spoofing_Warning | 18:44 |
| mordred | zaro: it would be neat if the boxes in the matrix matched the order from the config file, tbh | 18:44 |
| anteaya | sorry, https://wiki.openstack.org/wiki/GerritUpgrade | 18:44 |
| jaypipes | danke. | 18:44 |
| jeblair | (btw, acl updates have been running for a few mins) | 18:44 |
| mordred | jeblair: I'm STILL adding CLA users | 18:44 |
| zaro | mordred: boxes in the matrix? | 18:45 |
| zaro | mordred: what is that? | 18:45 |
| mordred | zaro: the thing you guys were talking about when I was talking about the wrong thing | 18:46 |
| *** nati_ueno has joined #openstack-infra | 18:46 | |
| *** yfried has quit IRC | 18:46 | |
| zaro | mordred: ohh yeah, agreed. | 18:46 |
| anteaya | jeblair: is line 27 done or still in progress? https://etherpad.openstack.org/p/J8WNqyP2dC | 18:46 |
| jeblair | anteaya: just updated | 18:46 |
| anteaya | k | 18:46 |
| anteaya | here is fungi's gpg host key fingerprint announcement for review: https://etherpad.openstack.org/p/fOMmg41vQ7 | 18:47 |
| jeblair | i'm going to stop zuul, run puppet to get the config change, then start it again | 18:47 |
| mordred | kk | 18:48 |
| morganfainberg | anteaya, that reads well and is straightforward. | 18:49 |
| anteaya | morganfainberg: thanks for looking | 18:49 |
| dims | new key looks good (verified using "ssh -p 29418 dims-v@review.openstack.org gerrit ls-projects") | 18:50 |
| *** _nadya_ has joined #openstack-infra | 18:50 | |
| jeblair | zuul is running | 18:51 |
| anteaya | dims: thanks for verifying | 18:51 |
| jeblair | manage-projects is finished | 18:51 |
| *** nati_ueno has quit IRC | 18:51 | |
| jeblair | spot checking project acls under https://review.openstack.org/#/admin/projects/ looks good | 18:52 |
| jeblair | mordred: how's the cla thing coming? | 18:52 |
| dims | anteaya, bonus for me i can get to it using ipv6 as well | 18:52 |
| anteaya | w00t | 18:52 |
| mordred | jeblair: still running | 18:53 |
| *** ativelkov has joined #openstack-infra | 18:53 | |
| fungi | anteaya: i updated the known_hosts entry removal command in the wiki since it needs the port number (based on my limited testing) | 18:54 |
| anteaya | fungi: thanks | 18:54 |
| *** msabramo has quit IRC | 18:54 | |
| jeblair | i don't think zuul works | 18:54 |
| fungi | anteaya: fwiw that's also roughly the same syntax openssh is recommending at me when i test connecting | 18:54 |
| jeblair | 2014-04-28 18:53:48,506 DEBUG zuul.DependentPipelineManager: Change <Change 0x31fcfd0 73469,2> can not merge, ignoring | 18:54 |
| jeblair | 2014-04-28 18:53:48,506 DEBUG zuul.DependentPipelineManager: Change <Change 0x31fcfd0 73469,2> is not ready to be enqueued, ignoring | 18:54 |
| anteaya | fungi: great, bonus for consistentcy | 18:55 |
| jeblair | oh | 18:55 |
| jeblair | it has no workflow +1 vote | 18:55 |
| jeblair | that sort of makes sense | 18:55 |
| jeblair | fungi: can you add approved votes to the following changes? | 18:56 |
| jeblair | fungi: 73469 86542 90660 90514 89277 89240 90086 90089 67657 83028 89484 86497 90380 84648 90396 | 18:56 |
| fungi | jeblair: on it now | 18:56 |
| jeblair | apparently the mergers are all failing too, i'm looking at that | 18:58 |
| jeblair | oh, heh | 18:58 |
| jeblair | they need their host keys updated :) | 18:59 |
| jeblair | their known_hosts, that is | 18:59 |
| mordred | hehe | 18:59 |
| *** praneshp has quit IRC | 18:59 | |
| mordred | jeblair: want help with that? | 18:59 |
| jeblair | mordred: i got it | 18:59 |
| mordred | k | 18:59 |
| jeblair | mordred: i wonder if anything else will..3. | 18:59 |
| mordred | gerritbot? | 19:00 |
| jeblair | mordred: yep | 19:01 |
| jeblair | well | 19:01 |
| jeblair | actually | 19:01 |
| jeblair | since it runs on review.o.o, probably not | 19:01 |
| jeblair | (we still need to fix that) | 19:01 |
| mordred | ++ | 19:01 |
| mordred | jeblair: it runs as gerrit user? | 19:02 |
| jeblair | maybe? | 19:02 |
| mordred | gerritbot user | 19:02 |
| mordred | I'm just going to go check | 19:02 |
| fungi | it may need it, yes | 19:02 |
| mordred | no - gerrit2 user | 19:03 |
| jeblair | it's on a new host, so it should not have a known_hosts file | 19:03 |
| fungi | also, approvals added to the list of changes from the previous gate queues | 19:03 |
| *** praneshp has joined #openstack-infra | 19:03 | |
| *** ativelkov is now known as ativelkov_away | 19:03 | |
| fungi | jeblair: oh, good point. it won't need the old key removed, but may need the new one accepted | 19:03 |
| mordred | just did the acceptance | 19:03 |
| *** ativelkov_away is now known as ativelkov | 19:03 | |
| jeblair | mordred: would you like to stop it on old review and start it on new review? | 19:03 |
| mordred | jeblair: yes I would | 19:04 |
| *** dkehn__ is now known as dkehnx | 19:04 | |
| *** openstackgerrit has quit IRC | 19:04 | |
| dansmith | is new gerrit expected to be faster, or is it just experiencing low load after being down for a bit? | 19:04 |
| zaro | mordred: i don't have access to dashboards? | 19:04 |
| mordred | dansmith: possibly both | 19:04 |
| dansmith | okay | 19:04 |
| zaro | mordred: did you already create the imporant changes dashboard? | 19:05 |
| mordred | dansmith: it's on a better server | 19:05 |
| dansmith | if it's faster, I might stop bitching about this +W business :) | 19:05 |
| *** changbl has quit IRC | 19:05 | |
| mordred | zaro: I did | 19:05 |
| *** changbl has joined #openstack-infra | 19:05 | |
| zaro | mordred: must be an acl setting. | 19:05 |
| mordred | dansmith: until you said that - I totally did not realize we were changing a communication semantic | 19:05 |
| dansmith | mordred: yeah, it's pretty unfortunate actually | 19:05 |
| mordred | I'm still going to say +A btw | 19:05 |
| dansmith | mordred: +A is in our vernacular | 19:06 |
| *** alexpilotti_ has joined #openstack-infra | 19:06 | |
| mordred | jeblair: I started gerritbot but he hasn't joined here | 19:06 |
| mordred | jeblair: IOError: [Errno 2] No such file or directory: '/home/gerrit2/.ssh/gerritbot_rsa' | 19:06 |
| jeblair | the value is still "approved" even though the column is "workflow". +A still makes sense to me | 19:07 |
| fungi | must not have puppeted that account's keys | 19:07 |
| *** alexpilotti has quit IRC | 19:07 | |
| *** alexpilotti_ is now known as alexpilotti | 19:07 | |
| mordred | in fact - we could say +A and -W now | 19:07 |
| jeblair | zuul host keys updated | 19:08 |
| fungi | i'll get started inserting the gerritbot rsa key into hiera | 19:08 |
| jeblair | and it's running tests | 19:08 |
| jeblair | fungi: thx | 19:08 |
| jeblair | i'm going to re-enqueue now | 19:08 |
| *** nati_ueno has joined #openstack-infra | 19:08 | |
| anteaya | mordred: is the cla thing finished? | 19:08 |
| *** ativelkov is now known as ativelkov_away | 19:09 | |
| mordred | fungi: yup. gerritbot ssh key not in hiera | 19:09 |
| zaro | mordred: ohh i found the dashboard. had to click on 'all-projects' first | 19:09 |
| * mordred putting in hiera - patch coming | 19:09 | |
| jeblair | i've restored the gate jobs | 19:09 |
| fungi | mordred: okay, all yours then | 19:09 |
| *** bookwar has left #openstack-infra | 19:09 | |
| jeblair | i'm not entirely sure i want to restore the check jobs right now | 19:09 |
| anteaya | I confirm the top 3 gate jobs are the same as when we stopped | 19:09 |
| anteaya | I'm guessing the rest are good too | 19:10 |
| fungi | Starting gate jobs. http://status.openstack.org/zuul/ | 19:10 |
| fungi | awesome | 19:10 |
| jeblair | it looks like the commentlink stuff that makes job results pretty is broken | 19:10 |
| anteaya | yeah, there were 175 in check when we stopped | 19:10 |
| *** sdake_ has quit IRC | 19:11 | |
| dims | folks, clicking on "HTTP Password" i see a HTTP 404 in firebug (https://review.openstack.org/accounts/self/password.http) | 19:11 |
| jeblair | dims: https://review.openstack.org/#/settings/http-password works for me | 19:11 |
| jeblair | dims: where did your url come from? | 19:12 |
| fungi | dims: neat. we don't use http passwords for gerrit, so should probably figure out how to disable that | 19:12 |
| jeblair | fungi: we do now -- they are required for rest api usage | 19:12 |
| fungi | oh! | 19:12 |
| dims | jeblair, click on the page with firebug enabled, you will see it | 19:13 |
| * fungi stands corrected | 19:13 | |
| *** esker has quit IRC | 19:13 | |
| anteaya | fungi: I didn't know that either | 19:13 |
| dims | the page itself loads fine | 19:13 |
| fungi | also, the generate password and clear password buttons are working for me at least | 19:13 |
| *** esker has joined #openstack-infra | 19:13 | |
| jeblair | dims: oh neat, it's an ajax thing | 19:14 |
| jeblair | dims: that works for me | 19:14 |
| jeblair | dims: do you have a 'generate password' or 'clear password' button? | 19:14 |
| *** gokrokve has joined #openstack-infra | 19:14 | |
| anteaya | dims: I don't know if the cla script has finished yet, what happens if you log out and back in again? | 19:14 |
| dims | i do see a "generate password" | 19:14 |
| jeblair | dims: click that? | 19:15 |
| *** nati_uen_ has joined #openstack-infra | 19:15 | |
| jeblair | dims: i don't think you get one automatically | 19:15 |
| jeblair | dims: (i think it works more like an api key) | 19:15 |
| YorikSar | dims: You get 404 because you have no password. It's REST | 19:15 |
| mriedem | are the previously marked WIP patches supposed to still show up under "my changes"? | 19:15 |
| dims | jeblair, PUT / DELETE works fine | 19:15 |
| mriedem | * my > changes | 19:15 |
| dims | YorikSar, cool that would do it | 19:15 |
| anteaya | mriedem: they should be workflow -1 | 19:16 |
| anteaya | so under W you should see a red X | 19:16 |
| fungi | oh, did we miss a step to update changes which were formerly wip to workflow=-1? | 19:16 |
| jeblair | anteaya: i do not believe there was any kind of data migration there | 19:16 |
| anteaya | jeblair: ohhh | 19:16 |
| anteaya | sorry | 19:16 |
| jeblair | anteaya: so all previously wip changes should now just be normal changes | 19:16 |
| mriedem | anteaya: not seeing them, only if i mark them as workflow=-1 after the change | 19:16 |
| anteaya | mriedem: I'm wrong | 19:16 |
| jeblair | fungi: i bet we could pull that list out of the old db | 19:16 |
| jeblair | fungi: want to do that? | 19:17 |
| jeblair | mordred: cla status? | 19:17 |
| mordred | jeblair: still running | 19:17 |
| jeblair | mordred: eta? | 19:17 |
| fungi | jeblair: yep, working on it now | 19:17 |
| anteaya | wow that takes a while | 19:17 |
| *** msabramo has joined #openstack-infra | 19:17 | |
| *** nati_ueno has quit IRC | 19:17 | |
| fungi | also, the puppet fix is pushed up as https://review.openstack.org/90803 | 19:18 |
| anteaya | fwiw that doesn't resolve for me | 19:18 |
| anteaya | the dns must not be updated around me yet | 19:18 |
| mordred | jeblair, fungi: https://review.openstack.org/90804 | 19:19 |
| fungi | anteaya: you may need to stop and relaunch your browser session | 19:19 |
| *** gokrokve has quit IRC | 19:19 | |
| jeblair | zaro: the diff pages look VERY different on review and review-dev | 19:19 |
| anteaya | fungi: I can try that, or I might clear my cache first | 19:19 |
| mordred | jeblair: just did 11113 out of 11281 | 19:20 |
| *** ramashri has quit IRC | 19:20 | |
| mordred | jeblair: vim tells me 98% | 19:20 |
| anteaya | cleared cache, no difference | 19:20 |
| zaro | jeblair: possible there was some change to it on ver 2.8.4 | 19:21 |
| zaro | jeblair: review-dev is running 2.8.3 | 19:21 |
| mordred | jeblair: CLA jobs done | 19:21 |
| mordred | jeblair: there were some users it complained on | 19:21 |
| jeblair | oh, i think it has something to do with wide diffs | 19:21 |
| jeblair | zaro: nm for now | 19:21 |
| *** bauzas has joined #openstack-infra | 19:22 | |
| anteaya | fungi: yeah, that did it | 19:22 |
| jeblair | fungi, mordred: think we can sound the all-clear for now? then i can grab some food and we can continue to clean things up? | 19:23 |
| fungi | jeblair: sure. that's probably a good time for me to e-mail out the host key change announcement too | 19:23 |
| mordred | jeblair: yes. I think so - although if you could +2 this: https://review.openstack.org/#/c/90804/ | 19:23 |
| anteaya | huh, I have one WIP patch that doesn't show up on my changes page at all | 19:24 |
| mordred | I can get gerritbot back up | 19:24 |
| anteaya | "create a seafile instance" | 19:24 |
| anteaya | was sure it was WIP | 19:24 |
| mriedem | anteaya: yeah, same, sounds like fungi is working on that | 19:24 |
| jeblair | mordred: done | 19:24 |
| fungi | anteaya: yeah, i'm working on digging that out of the old db | 19:24 |
| anteaya | k, sorry | 19:25 |
| harlowja | hmmm, whats up with the fonts :-P | 19:25 |
| anteaya | thought they would still be in changes, just normal patches | 19:25 |
| anteaya | harlowja: can you be more specific? | 19:25 |
| harlowja | seems like commit messages have different font then the rest of the page now :) | 19:25 |
| harlowja | guess thats a feature :-P | 19:25 |
| mordred | harlowja: they're editable in browser now too - as are the topic | 19:26 |
| harlowja | intersting | 19:26 |
| fungi | i find in-browser commit message editing slightly weird | 19:26 |
| harlowja | hmmm, new-screen view also seems new | 19:27 |
| *** ramashri has joined #openstack-infra | 19:27 | |
| mordred | jeblair: thanks | 19:27 |
| * afazekas https://review.openstack.org/#/c/90782 I got 500 from js, (not important to fix) it was WIP change | 19:27 | |
| *** e0ne has quit IRC | 19:28 | |
| *** mrmartin has quit IRC | 19:28 | |
| morganfainberg | editing commit messages in the browser... thats... kinda cool | 19:28 |
| mordred | jeblair: hrm - having the comment report: "Patch Set 1: Code-Review+2 Workflow+1" will take a little getting used to | 19:28 |
| jeblair | ok Gerrit upgradet to 2.8 complete. See: https://wiki.openstack.org/wiki/GerritUpgrade Some cleanup tasks still ongoing; join #openstack-infra if you have any questions. | 19:28 |
| dkranz | fungi: I see links in jenkins comments now show full url (2 lines per entry instead of 1). Is that expected? | 19:28 |
| mordred | jeblair: ++ | 19:28 |
| harlowja | do u guys know if the ssh stream that gerrit emits is changed much by this, might have to update my tool :-P | 19:28 |
| jeblair | mordred, fungi, anteaya: ^ look good ? | 19:28 |
| jeblair | harlowja: not much; zuul still works | 19:29 |
| anteaya | jeblair: yes | 19:29 |
| harlowja | k | 19:29 |
| *** mrmartin has joined #openstack-infra | 19:29 | |
| jeblair | #status ok Gerrit upgrade to 2.8 complete. See: https://wiki.openstack.org/wiki/GerritUpgrade Some cleanup tasks still ongoing; join #openstack-infra if you have any questions. | 19:29 |
| openstackstatus | jeblair: sending ok | 19:29 |
| fungi | jeblair: lgtm | 19:29 |
| jeblair | whew | 19:29 |
| anteaya | harlowja: stream-events now needs express permission, it isn't available to registered users anymore | 19:30 |
| fungi | dkranz: it needs some work. something about the commentlinks configuration syntax probably changed | 19:30 |
| jeblair | anteaya: yes it is | 19:30 |
| anteaya | is it? | 19:30 |
| harlowja | phew | 19:30 |
| anteaya | I'm wrong | 19:30 |
| YorikSar | jeblair: Will it be a good time now to ask to add one line to project.config? | 19:30 |
| jeblair | YorikSar: no | 19:30 |
| fungi | anteaya: it isn't by default on a plain gerrit install, so we added it to the default in our config | 19:31 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 19:31 | |
| -openstackstatus- NOTICE: Gerrit upgrade to 2.8 complete. See: https://wiki.openstack.org/wiki/GerritUpgrade Some cleanup tasks still ongoing; join #openstack-infra if you have any questions. | 19:31 | |
| bnemec | Hmm, getting an Internal Server Error trying to look at https://review.openstack.org/#/c/81146/ | 19:31 |
| anteaya | I connected those dots, didn't want to add to noise atm though | 19:31 |
| jeblair | YorikSar: we still have quite a bit of work to do; you're asking for a behavioral change. that will require quite a bit of discussion which would be distracting now. | 19:31 |
| anteaya | bnemec: can you restart your browser and try again? | 19:32 |
| jeblair | anteaya: does bnemec's change work for you? | 19:32 |
| *** gabriel-bezerra has joined #openstack-infra | 19:32 | |
| anteaya | ah no it doesn't | 19:32 |
| morganfainberg | jeblair, i get ISE on bnemec's change | 19:32 |
| bnemec | I get the same thing from afazekas's link above. | 19:32 |
| anteaya | sorry, I was dispensing wrong advice | 19:32 |
| *** ashishsnigam has joined #openstack-infra | 19:32 | |
| YorikSar | jeblair: Do those changes get applied without Gerrit restart? | 19:32 |
| jeblair | YorikSar: what changes? | 19:33 |
| openstackstatus | jeblair: finished sending ok | 19:33 |
| bknudson | 503 Service Temporarily Unavailable | 19:33 |
| jeblair | fungi, mordred: we need to accept replication host keys for the gerrit user | 19:33 |
| YorikSar | jeblair: I mean changes to project.config that affect labels behavior. | 19:33 |
| anteaya | bnemec: which url is afazekas's link? | 19:33 |
| mordred | jeblair: k. I can do that | 19:33 |
| bnemec | anteaya: https://review.openstack.org/#/c/90782 | 19:33 |
| jeblair | YorikSar: please let's discuss this later. | 19:33 |
| anteaya | bknudson: have you restarted your browser? | 19:33 |
| YorikSar | jeblair: Ok. Sure. | 19:33 |
| fungi | mordred: jeblair: and then re-trigger replication afterward | 19:34 |
| bnemec | anteaya: I wonder if it's a WIP thing. | 19:34 |
| bknudson | anteaya: nope, will do. | 19:34 |
| mordred | jeblair: next time you get a sec - I could use help with debugging gerritbot | 19:34 |
| anteaya | bnemec: is your patch a WIP? | 19:34 |
| jeblair | mordred: i think i need to get food right now | 19:34 |
| psedlak | it seems WIP changes are lost | 19:34 |
| bnemec | anteaya: Not sure, it isn't my patch. | 19:34 |
| anteaya | bnemec: if yes, then yes, WIP patches were not carried over in the migration | 19:34 |
| mordred | it seems unhappy with irc | 19:34 |
| afazekas | anteaya: I guess the WIP changes has an upgrade issue | 19:34 |
| anteaya | psedlak: yes | 19:34 |
| mordred | jeblair: ok - I'll keep poking - get food | 19:34 |
| anteaya | fungi is retrieving WIP patches from old gerrit | 19:34 |
| *** sdake_ has joined #openstack-infra | 19:34 | |
| jeblair | afazekas, bnemec: oh were the erroring changes marked wip before? | 19:35 |
| anteaya | since the WIP behaviour is changed, we didn't know our migration wouldn't bring them across | 19:35 |
| anteaya | fungi is getting them | 19:35 |
| jeblair | i wonder if the problem is that they still have that state in the db which is unknown to gerrit | 19:35 |
| fungi | anteaya: i am in the process of retrieving the list of which changes were marked wip so we can add a workflow -1 on them | 19:35 |
| anteaya | fungi: thanks | 19:35 |
| fungi | anteaya: sounds like the issue may run deeper than that though | 19:35 |
| * anteaya stands by | 19:35 | |
| bnemec | Yeah, probably WIP. I'll try again once those have been brought over. | 19:35 |
| afazekas | jeblair: y | 19:35 |
| anteaya | bnemec: thanks | 19:36 |
| morganfainberg | bnemec, yeah i noticed a bunch of reviews seem to be gone from Keystone as well. Likely WIP ones | 19:36 |
| jeblair | fungi: okay, so we probably need to manually update the status on those to 'new' in the new db. | 19:36 |
| bknudson | what's the status field for? | 19:36 |
| anteaya | bknudson: what are you looking at? | 19:36 |
| *** eharney has quit IRC | 19:36 | |
| bknudson | anteaya: https://review.openstack.org/#/q/status:open,n,z | 19:37 |
| *** ChanServ sets mode: +o jeblair | 19:37 | |
| fungi | jeblair: interesting. i'll figure out what you mean by that once i poke around in the new db schema, i'm sure | 19:37 |
| boris-42 | don't touch WIP button lol | 19:37 |
| anteaya | bknudson: right now I don't know | 19:37 |
| jeblair | fungi: zaro or mordred may be able to explain | 19:37 |
| anteaya | status is open, merged, abandoned | 19:37 |
| anteaya | bknudson: in the url status is open | 19:38 |
| bknudson | ok, abandoned and merged show up on those views | 19:38 |
| *** jeblair changes topic to "Known problem with WIP changes. Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade" | 19:38 | |
| anteaya | bknudson: https://review.openstack.org/#/q/status:merged,n,z | 19:38 |
| zaro | fungi: the WIP feature added a wip status. that's no longer a valid status in the db | 19:38 |
| hashar | congratulations folks! | 19:38 |
| *** nati_uen_ has quit IRC | 19:38 | |
| anteaya | hashar: thanks | 19:38 |
| * jeblair will grab food quickly and return | 19:38 | |
| anteaya | jeblair: /nod | 19:39 |
| anteaya | still some small fires | 19:39 |
| mordred | ok. replication kicked | 19:39 |
| hashar | I know how long you spent preparing that upgrade. Seems it went fine (at least I can see my dashboard) | 19:39 |
| anteaya | hashar: a happy customer | 19:39 |
| anteaya | thank you | 19:39 |
| mordred | hashar: thanks! big ticket items done at least- still some cleanup to go | 19:39 |
| anteaya | zaro spent the most time on the upgrade | 19:39 |
| fungi | zaro: okay, so i may just be able to add a wip vote for any changes in an (invalid) wip status and reset the status to new at the same time | 19:39 |
| bknudson | I think I had a few WIP patches. No big deal if they're gone | 19:39 |
| anteaya | bknudson: k, we are working on getting them back | 19:40 |
| zaro | fungi: ++ | 19:40 |
| fungi | bknudson: looks like they're still in the db, just need to be adjusted | 19:40 |
| hashar | mordred: you will love 2.8 :-) | 19:40 |
| *** nati_uen_ has joined #openstack-infra | 19:40 | |
| *** thomasbiege has quit IRC | 19:40 | |
| *** chandan_kumar has quit IRC | 19:40 | |
| bknudson | https://review.openstack.org/#/c/66137/ -- says Internal server error | 19:40 |
| anteaya | bknudson: that might be a WIP patch url | 19:41 |
| fungi | bknudson: yep | 19:41 |
| anteaya | do you know? | 19:41 |
| *** matrohon has quit IRC | 19:41 | |
| *** pelix has joined #openstack-infra | 19:42 | |
| bknudson | anteaya: I think it might depend on a wip? | 19:43 |
| anteaya | bknudson: that might explain it | 19:43 |
| bknudson | anteaya: and I don't see that other review in my list | 19:43 |
| *** ramashri has quit IRC | 19:43 | |
| anteaya | bknudson: that other review? | 19:43 |
| anteaya | the wip review | 19:44 |
| *** openstackgerrit has joined #openstack-infra | 19:44 | |
| YorikSar | That link works with new change screen. | 19:44 |
| YorikSar | Fails for Old screen. | 19:44 |
| *** jamespd has joined #openstack-infra | 19:44 | |
| *** openstackgerrit has quit IRC | 19:44 | |
| bknudson | anteaya: I stuck another review in front of it over the weekend... maybe I can find the link | 19:44 |
| YorikSar | (new change screen is cool though) | 19:45 |
| anteaya | bknudson: well if there is a chain with a wip in there, might better make a cup of tea and let fungi move those wip patches over | 19:45 |
| YorikSar | bknudson: @Fix tests to use strings rather than ints for IDs" - thix one? | 19:45 |
| bknudson | I'm not in a hurry | 19:45 |
| anteaya | bknudson: since this all might be different colours of the same issue, missing wip patches | 19:45 |
| bknudson | YorikSar: that's the one I'm thinking of | 19:45 |
| anteaya | bknudson: k, thanks | 19:45 |
| YorikSar | bknudson: https://review.openstack.org/#/c/90621/ | 19:46 |
| YorikSar | Fails even in new change screen... | 19:46 |
| *** ociuhandu has quit IRC | 19:47 | |
| mordred | jeblair: for when you get back - gerritbot seems quite happy talking to new gerrit - but is having problems talking to IRC | 19:47 |
| bknudson | I've got the changes locally so no big deal if they're gone. | 19:48 |
| anteaya | bknudson: k | 19:48 |
| anteaya | mordred: did add CLA users ever finish? https://etherpad.openstack.org/p/J8WNqyP2dC | 19:49 |
| YorikSar | bknudson: They are definitelly not gone - they are accessible through Git. | 19:50 |
| *** lcostantino has quit IRC | 19:51 | |
| *** nati_uen_ has quit IRC | 19:51 | |
| fungi | and still in the db | 19:51 |
| * anteaya nods | 19:51 | |
| anteaya | go you | 19:52 |
| *** _nadya_ has quit IRC | 19:52 | |
| mordred | fungi, jeblair: https://review.openstack.org/90807 | 19:53 |
| mordred | that encompasses the changes jeblair made on the filesystem | 19:53 |
| mordred | well, that and fungi's change | 19:53 |
| *** dprince has quit IRC | 19:53 | |
| *** mrmartin has quit IRC | 19:53 | |
| *** sweston has quit IRC | 19:54 | |
| *** nati_ueno has joined #openstack-infra | 19:55 | |
| *** mrmartin has joined #openstack-infra | 19:55 | |
| fungi | okay, we're in luck. changes with status="w" are all also open="Y" so i'm making a list (so that i'll know which ones to add a wip vote on) and then updating them to status="n" | 19:56 |
| anteaya | fungi: yay | 19:56 |
| *** sarob has joined #openstack-infra | 19:56 | |
| anteaya | you can add a wip vote on them via ssh, yes? | 19:56 |
| anteaya | or via the db | 19:56 |
| fungi | if i give myself permission to do so, yes via ssh | 19:56 |
| anteaya | great | 19:57 |
| fungi | voting via the db is more complicated since it would involve touching multiple tables, i think | 19:57 |
| mordred | the important changes screen is not showing the right things for previously reviewed | 19:57 |
| anteaya | fungi: let's do the less complicated thing | 19:57 |
| anteaya | mordred: what it is missing or adding? | 19:58 |
| anteaya | mordred: and is add CLA users finished running? | 20:00 |
| mordred | anteaya: it's showing me things that I have not reviewed before, nor am subscribed to | 20:00 |
| YorikSar | fungi: You can also suexec to the owner of a change to vote for it. | 20:00 |
| anteaya | mordred: that's odd | 20:00 |
| *** atiwari has joined #openstack-infra | 20:00 | |
| *** annegentle has quit IRC | 20:01 | |
| *** matty_dubs has joined #openstack-infra | 20:01 | |
| anteaya | yeah me too, I have https://review.openstack.org/#/c/77434/ under important-changes, changes needing review | 20:02 |
| anteaya | never seen it before | 20:02 |
| anteaya | oh hang on there's my name, I have so | 20:02 |
| fungi | YorikSar: yeah, that involves authenticating to gerrit with its own host key and the syntax on suexec is a little messy | 20:02 |
| anteaya | I did a recheck no bug | 20:02 |
| anteaya | no, changes needing review in my important changes dashboard all have my name on them | 20:04 |
| YorikSar | fungi: I thought you only need "Run As" right for that... | 20:04 |
| jogo | oh neat edit commit message | 20:05 |
| fungi | YorikSar: oh, maybe. in earlier versions you had to use the special "gerrit code review" account, but i'm reading the new api docs now | 20:06 |
| *** jgrimm has quit IRC | 20:06 | |
| fungi | YorikSar: indeed you are correct... "suexec can only be invoked by the magic user Gerrit Code Review, or any user granted granted the Run As capability." | 20:06 |
| *** nati_ueno has quit IRC | 20:06 | |
| anteaya | but reviewed changes all have the most recent timestamp, irrespective if I have seen them before or not | 20:06 |
| fungi | YorikSar: though also seems to require additional configuration | 20:07 |
| jeblair | fungi: if you want to grant that to project bootstrappers i'd be okay with that | 20:07 |
| YorikSar | fungi: I guess superadmins (infra-core) can have "Run As" right just in case. | 20:07 |
| fungi | "The run as capability is permitted to be used only if auth.enableRunAs is true." https://review.openstack.org/Documentation/cmd-suexec.html | 20:07 |
| anteaya | and I can understand the error, since when clarkb was testing the dashboard the list of recent changes was small and he had probably reviewed them all | 20:07 |
| jeblair | oh | 20:07 |
| fungi | so maybe another time | 20:07 |
| *** denis_makogon has quit IRC | 20:09 | |
| fungi | testing now to see if a workflow -1 vote i leave while a project bootstrapper on a change i wouldn't otherwise control in that regard sticks after removing myself from the group, and it seems to | 20:10 |
| fungi | afazekas: can you test un-wip'ing https://review.openstack.org/90782 now that i've set it to workflow -1? | 20:11 |
| anteaya | mordred: at one point this was the query clarkb pushed for important changes: http://paste.openstack.org/show/76693/ | 20:11 |
| fungi | i just want to make sure that i'm not about to leave a bunch of wip votes which only admins can remove ;) | 20:11 |
| anteaya | mordred: I am not sure if there was editing or not | 20:11 |
| *** jreypo has joined #openstack-infra | 20:11 | |
| zaro | line number on right side on change screen is annoying :( | 20:11 |
| zaro | opps i mean diff screeen | 20:12 |
| anteaya | yes, it is a small change though | 20:12 |
| anteaya | we can get used to it | 20:12 |
| jeblair | anteaya: speak for yourself | 20:13 |
| anteaya | really? | 20:13 |
| anteaya | okay, it would be something I would just say meh to | 20:13 |
| zaro | on master version of gerrit, it got put back to left side. | 20:14 |
| *** gokrokve has joined #openstack-infra | 20:14 | |
| *** adalbas has quit IRC | 20:14 | |
| mordred | anteaya: ok. I've updated the important changes dashboard with that | 20:14 |
| jeblair | mordred: what did you change? | 20:14 |
| jeblair | mordred: also, still need me to look at gerritbot? | 20:15 |
| YorikSar | fungi: You can test on one of my changes. E.g. https://review.openstack.org/#/c/81798/ | 20:15 |
| anteaya | mordred: no no, I had said that was one of the versions clark had suggested | 20:15 |
| mordred | jeblair: yes please - | 20:15 |
| mordred | anteaya: well, it did not help :) | 20:15 |
| anteaya | I wasn't saying that was the fix | 20:15 |
| * mordred will go dig in to review-dev and see what's there | 20:15 | |
| anteaya | my comment was that reviewed changes just show the most recent changes in gerrit | 20:15 |
| mordred | jeblair: the debug log says things about joining channels, but I do not see that happening | 20:16 |
| fungi | YorikSar: done. are you able to un-wip that now? | 20:16 |
| anteaya | and that clarkb probably made a logical mistake since he was working with review-dev, not many changes on it | 20:16 |
| *** gokrokve_ has joined #openstack-infra | 20:16 | |
| jeblair | zaro: the trivial rebase hook is running, was that supposed to be disabled? | 20:16 |
| YorikSar | fungi: Nope | 20:17 |
| zaro | jeblair: yes, i believe so. | 20:17 |
| YorikSar | fungi: Every user have one's own labels. | 20:17 |
| * psedlak is not happy about clicking the project name or that 'cool-fluffy-icon' thing next to it while looking at single change ... bringing me to the wierd web-form or to the view with all open, abandoned and merged changes ... really ... no direct link to open changes ... ok bookmarking wins | 20:17 | |
| fungi | YorikSar: that's an interesting drawback to the new workflow design | 20:17 |
| YorikSar | fungi: So only you can remove your -1. | 20:17 |
| YorikSar | fungi: Well... Only owner is supposed to wip and un-wip. | 20:17 |
| fungi | YorikSar: we granted core reviewers wip control with this too | 20:18 |
| fungi | YorikSar: we might want to re-think that | 20:18 |
| YorikSar | fungi: Ah... Yeah. | 20:18 |
| jeblair | fungi: users can upload a new patchset to clear wip reviews | 20:18 |
| fungi | jeblair: ahh, fair enough! | 20:18 |
| *** ramashri has joined #openstack-infra | 20:18 | |
| jeblair | fungi: that's an intentional balance to the design of allowing core to -1 wip | 20:19 |
| *** gokrokve has quit IRC | 20:19 | |
| fungi | jeblair: which then makes a good argument for not having work-in-progress convey to subsequent patchets | 20:19 |
| YorikSar | jeblair, fungi: the feature users want to get rid of for a long time :) | 20:19 |
| YorikSar | I guess we'll have to patch git-review then. | 20:19 |
| afazekas | fungi: unwip - ed, it still gets a red X | 20:20 |
| jeblair | fungi: if you want to enable runas, we can restart gerrit for that | 20:20 |
| fungi | afazekas: yep, thanks | 20:20 |
| *** gokrokve_ has quit IRC | 20:20 | |
| zaro | jeblair: just checked again, trival rebase disabled in puppet looks ok to me. | 20:20 |
| anteaya | mordred: try this: http://paste.openstack.org/show/77526/ | 20:21 |
| fungi | jeblair: probably worth doing. i'll draft up the config change for it | 20:21 |
| zaro | jeblair: and it's not running on review-dev so puppet seems to not be running it there. | 20:21 |
| psedlak | afazekas: red X where? after unWIPing? | 20:21 |
| fungi | psedlak: if someone else marks your patchset as a work in progress, you can't clear it without uploading a new patchset | 20:21 |
| fungi | psedlak: though only the change owner or a core reviewer for the target project/branch can do that, so shouldn't be a huge deal | 20:22 |
| *** lcostantino has joined #openstack-infra | 20:22 | |
| anteaya | jeblair: I don't know the status of add CLA users | 20:23 |
| anteaya | jeblair: I am guessing it is finished but don't know for sure | 20:23 |
| jeblair | anteaya: check scrollback, mordred said it's done. | 20:23 |
| psedlak | fungi: thx, yeah while cores should not be doing it, it's better to not let them do it at all | 20:23 |
| mordred | jeblair, anteaya: ok. important changes dashboard should be fixed now | 20:24 |
| anteaya | jeblair: sorry, thanks, I will update the etherpad | 20:24 |
| psedlak | fungi: i mean only the owner should have the right to use W=-1 | 20:24 |
| anteaya | mordred: did you use my patch? | 20:24 |
| jeblair | psedlak: no, cores should have it. cores know when a change absolutely needs more work and no other cores should waste time reviewing it. that's what it's for. | 20:24 |
| mordred | NOT label:code-review=0,self does not work when there are projects that you don't watch (Which is what we missed on review-dev) | 20:24 |
| fungi | psedlak: well, you can also clear it with a new patchset (rebase, et cetera) or ask them to remove their workflow -1 vote | 20:24 |
| *** nati_ueno has joined #openstack-infra | 20:25 | |
| mordred | anteaya: oh - I did not see that before I made mine ... I think it still might miss something, although yours would certainly be better | 20:25 |
| mordred | anteaya: I wound up just doing this: status:open (label:code-review=-1,self OR label:code-review=-2,self OR label:code-review=+1,self OR label:code-review=+2,self) limit:50 | 20:25 |
| anteaya | replication kicked? I guess I didn't know what that meant | 20:25 |
| mordred | anteaya: I told replication to run again | 20:25 |
| psedlak | jeblair: well i think think WIP should be more personal, cores can use -1/2s for that anyway | 20:26 |
| dansmith | are things actually flusing from the check queue? | 20:26 |
| jeblair | psedlak: let's discuss this some other time | 20:26 |
| dansmith | looks like several are done, but aren't voting and dropping out | 20:26 |
| jeblair | dansmith: ack -- unknown problem. | 20:26 |
| *** adalbas has joined #openstack-infra | 20:26 | |
| jeblair | dansmith: thanks for reporting | 20:26 |
| dansmith | jeblair: okay | 20:26 |
| *** mwagner_lap has quit IRC | 20:27 | |
| dansmith | jeblair: did you mean you hadn't noticed, or you don't know why yet? | 20:27 |
| psedlak | fungi: but that's twisting the usage - if you want to clear someone else wip, just do bull.. rebase/touch-the-code-twice review upload ... that does not sounds ok | 20:27 |
| anteaya | mordred: okay, well if your important changes shows you want you are looking for | 20:27 |
| jeblair | dansmith: heh, sorry. i had not noticed; you are the first person to report. | 20:27 |
| psedlak | jeblair: sure, it's not a big issue | 20:27 |
| dansmith | woot! | 20:27 |
| dansmith | the growing backlog seems like maybe it indicates that the first thing to finish has deadlocked zuul or something | 20:28 |
| dansmith | like, if the vote failed or something | 20:29 |
| mordred | jeblair: we don't seem to be replicating to github - i'm investigating | 20:31 |
| jeblair | dansmith: something just changed | 20:31 |
| mordred | jeblair: I thinkw e need to restart gerrit | 20:32 |
| mordred | jeblair: it seems to cache ssh host keys and flush-caches does not fix it | 20:32 |
| jeblair | mordred: that's correct | 20:33 |
| mordred | it's re-trying ssh-ing to git* in a very tight loop right now | 20:33 |
| fungi | mordred: jeblair: if we're going to restart gerrit, let's add https://review.openstack.org/90814 (either force submit or hand-patch it on the server) | 20:33 |
| jeblair | fungi: have the runas change ready so it can go in with mordred's restart? | 20:33 |
| fungi | jeblair: that's the one | 20:33 |
| mordred | we're going to have to hand apply | 20:33 |
| mordred | gerrit isn't replicating, so we're not pulling new changes into puppet | 20:34 |
| jeblair | ++ | 20:34 |
| mordred | hand applying now | 20:34 |
| mordred | jeblair, fungi: I'm ready to restart now - you guys good with that? | 20:35 |
| *** bookwar has joined #openstack-infra | 20:35 | |
| jeblair | mordred: yep | 20:35 |
| mordred | doing | 20:35 |
| jeblair | zuul is getting a lot of errors from gerrit because of this: | 20:36 |
| jeblair | fatal: "--force-message" is not a valid option | 20:36 |
| fungi | mordred: ye[ | 20:36 |
| fungi | yep | 20:36 |
| jeblair | hashar: have you run into that ^ ? | 20:36 |
| hashar | oh men | 20:36 |
| hashar | yeah that got removed from gerrit review command | 20:36 |
| hashar | sorry should have thought about it | 20:36 |
| lifeless | review.o.o is 503ing right now | 20:36 |
| jeblair | lifeless: topic | 20:36 |
| fungi | lifeless: it's undergoing another restart | 20:36 |
| lifeless | that might be relatde to zuul having troubl | 20:36 |
| lifeless | jeblair: ack | 20:37 |
| mordred | restarted | 20:37 |
| hashar | jeblair: basically force-message is not needed any more. You can gerrit review "some message" on a closed change. | 20:37 |
| fungi | lifeless: cause<-effect ;) | 20:37 |
| lifeless | topic stops at GerritUpg for me, unless I explicitly check. | 20:37 |
| jeblair | hashar: what about reviewing old patchsets? | 20:37 |
| anteaya | https://wiki.openstack.org/wiki/GerritUpgrade | 20:37 |
| lifeless | jeblair: home page btw, not WIP changes... or I wouldn't hav ementioned it. | 20:37 |
| hashar | jeblair: same that should work out of the box. Let me find my layout.yaml tweak :) | 20:37 |
| *** sweston has joined #openstack-infra | 20:38 | |
| mordred | sigh. restarting once more - missed a hostkey that gerrit didn't have yet | 20:38 |
| hashar | jeblair: workaround is to pass verified: 0 https://gerrit.wikimedia.org/r/#/c/117893/1/layout.yaml,unified | 20:38 |
| jeblair | lifeless: oh, yeah, that's the restart then. sorry. | 20:38 |
| mordred | restarted | 20:38 |
| mordred | (much quicker that time) | 20:38 |
| hashar | jeblair: I think that is Zuul code requiring an action when sending a message. Or to say it otherwise, Zuul dont let you simply: gerrit review 1234,1 --message "hello world" | 20:39 |
| hashar | jeblair: it requires an action like verified / code-review | 20:39 |
| jeblair | hashar: i was thinking about when it votes verified=-1 on an old patchset, does that still work without force-message? | 20:39 |
| mordred | somethign tells me this is a lie: [2014-04-28 20:39:23,961] INFO com.googlesource.gerrit.plugins.replication.ReplicationQueue : Created remote repository: git@github.com:API-Projects.git | 20:39 |
| fungi | hah | 20:39 |
| jeblair | mordred: that would be cool :) | 20:40 |
| *** dkliban is now known as dkliban_afk | 20:40 | |
| hashar | jeblair: the verified:0 vote is discard by Gerrit since the job is closed, but Gerrit 2.8 will record the message as a comment | 20:40 |
| *** julim has quit IRC | 20:40 | |
| sweston | anteaya: jeblair: mordred: fungi: I had to step out for a few meetings .. I see you all are still at it? Anything I can do to help? Please let me know :-) | 20:40 |
| jeblair | hashar: yeah, verified=0 is fine, i'm concerned with verified=-1 on a previous patchset | 20:40 |
| *** bhuvan has quit IRC | 20:40 | |
| hashar | jeblair: isn't --force-message used on closed changes? | 20:41 |
| hashar | jeblair: once closed, the verify vote is ignored. So the verify-1 should stick | 20:41 |
| *** jgrimm has joined #openstack-infra | 20:41 | |
| jeblair | hashar: oh deal | 20:41 |
| jeblair | oh dear, i mean | 20:41 |
| hashar | you might want to try it out with your use cases though | 20:41 |
| jeblair | hashar: if we have to do verified=0, we can't have experimental queues anymore | 20:41 |
| jeblair | they _depend_ on leaving no votes whatsoever | 20:42 |
| *** bhuvan has joined #openstack-infra | 20:42 | |
| hashar | so just a message | 20:42 |
| hashar | which Gerrit let you do, but Zuul reject due to some logic madness | 20:42 |
| * hashar digs | 20:43 | |
| jeblair | hashar: yeah, it's so that there's _some_ action there | 20:44 |
| hashar | jeblair: yeah that is in zuul/reporter/gerrit.py Reporter.report() will returns early if it is missing a params | 20:44 |
| jeblair | hashar: i guess we'll have to patch zuul to no longer require that | 20:44 |
| hashar | (the function takes params and message) | 20:44 |
| hashar | so if you pass it only a message (and no params), the message is never sent | 20:44 |
| hashar | sorry should have left a note on the ether pad :( completely forgot about that one | 20:45 |
| hashar | the pity is that I fixed it only a month and a half ago | 20:45 |
| jeblair | hashar: do you mean you fixed zuul? | 20:45 |
| jeblair | hashar: or you changed your layout.yaml? | 20:45 |
| hashar | na i fixed the layout by passing verified: 0 | 20:45 |
| jeblair | k | 20:45 |
| hashar | this way there is a param passed to Gerrit reporter | 20:46 |
| fungi | does gerrit 2.8 now solve the issue with comments and no associated vote not counting as review activity? | 20:46 |
| hashar | and it executes the command | 20:46 |
| greghaynes | Interesting upgrade bug - people with quotes in their displayname cause ISE when clicking to go to their dashboard. e.g. https://review.openstack.org/#/q/owner:%22Clint+%22SpamapS%22+Byrum%22+status:open,n,z | 20:46 |
| *** eharney has joined #openstack-infra | 20:46 | |
| hashar | jeblair: with Gerrit 2.4 that would have rejected the command because you can't vote, hence why we had to pass force-message | 20:46 |
| anteaya | greghaynes: thanks | 20:47 |
| *** nati_ueno has quit IRC | 20:47 | |
| jeblair | hashar: i think the check in there dates from an ealier syntax in layout.yaml. where the lack of params was an indication that it should be silent.... | 20:48 |
| jeblair | hashar: but now we have explicit reporters, so i think the lack of _reporter_ indicates the same thing | 20:48 |
| *** ociuhandu has joined #openstack-infra | 20:48 | |
| hashar | I guess:) | 20:48 |
| jeblair | hashar: so i think we should be able to say that if there's a reporter configured, even with no params, it should leave a msg | 20:48 |
| *** afazekas has quit IRC | 20:48 | |
| jeblair | mordred: (i've dropped the irc thing in favor of zuul) | 20:49 |
| mordred | jeblair: yes. I agree with you | 20:49 |
| anteaya | zaro: click Clint's display name: https://review.openstack.org/#/c/90515/ | 20:49 |
| SpamapS | I _must_ have my quotes. | 20:49 |
| SpamapS | AND YES | 20:49 |
| mordred | jeblair: it's my turn to need to step away for a sandwich | 20:49 |
| anteaya | zaro: how do we fix that | 20:49 |
| SpamapS | I found another way to break Gerrit!! | 20:49 |
| anteaya | SpamapS: well done | 20:49 |
| anteaya | SpamapS: working on it | 20:49 |
| SpamapS | Gerrit: pwned | 20:50 |
| mordred | wtm | 20:50 |
| mordred | SpamapS: what is it that doesn't work? | 20:50 |
| anteaya | https://review.openstack.org/#/c/90515/ | 20:50 |
| anteaya | click his name | 20:50 |
| mordred | yup | 20:50 |
| anteaya | internal server erro | 20:50 |
| anteaya | r | 20:50 |
| mordred | that gets me: https://review.openstack.org/#/q/owner:clint%2540ubuntu.com+status:open,n,z | 20:50 |
| zaro | anteaya: fix what? | 20:50 |
| mordred | which works fine | 20:50 |
| mordred | NOW | 20:50 |
| zaro | wfm too | 20:50 |
| mordred | if I click his name from that listing | 20:51 |
| mordred | I get failure | 20:51 |
| anteaya | that | 20:51 |
| hashar | jeblair: got a patch to Gerrit reporter with some background in the commit summart | 20:51 |
| hashar | summary | 20:51 |
| jeblair | hashar: link? | 20:52 |
| *** SumitNaiksatam has joined #openstack-infra | 20:52 | |
| anteaya | did 2.4.4 use quotes around owner? https://review.openstack.org/#/q/owner:%22Monty+Taylor%22+status:open,n,z | 20:52 |
| anteaya | I didn't think it did | 20:53 |
| *** sarob has quit IRC | 20:54 | |
| fungi | YorikSar: success! https://review.openstack.org/81798 | 20:54 |
| mordred | SpamapS: can you try changing your double quotes to single quotes? | 20:54 |
| *** marcoemorais has quit IRC | 20:54 | |
| fungi | now i just need to work on querying the latest patchset numbers and owner e-mails for the ~300 wip changes | 20:55 |
| YorikSar | fungi: Oh. I've put my -1 there now? Really? :) | 20:55 |
| YorikSar | fungi: unwiping works fine now | 20:56 |
| fungi | YorikSar: excellent, that's what i was hoping | 20:56 |
| ArxCruz | I'm getting BadHostKeyException: Host key for server review.openstack.org does not match! in my zuul | 20:56 |
| *** nati_ueno has joined #openstack-infra | 20:56 | |
| *** marcoemorais has joined #openstack-infra | 20:56 | |
| anteaya | ArxCruz: https://wiki.openstack.org/wiki/GerritUpgrade#DNS_Spoofing_Warning | 20:56 |
| YorikSar | fungi: You can use new shiny REST API. | 20:56 |
| fungi | ArxCruz: the user under which your zuul daemon is running probably needs to clear the old gerrit host key from known_hosts | 20:57 |
| hashar | jeblair: sorry https://review.openstack.org/90816 | 20:57 |
| ArxCruz | fungi: anteaya ty guys :) | 20:57 |
| hashar | jeblair: was fighting with git-review :D | 20:57 |
| fungi | YorikSar: good idea | 20:57 |
| anteaya | ArxCruz: let us know, you are the first of many 3rd party ci systems that will show up | 20:58 |
| *** Sukhdev has joined #openstack-infra | 20:58 | |
| anteaya | ArxCruz: would be good to have your feedback | 20:58 |
| SpamapS | mordred: sure | 20:58 |
| ArxCruz | anteaya: sure :) | 20:58 |
| anteaya | ArxCruz: thanks | 20:58 |
| hashar | jeblair: no test though :-( | 20:58 |
| YorikSar | fungi: It won't work with jq though.. You need to skip first 4 chars in every response to get parseable JSON | 20:58 |
| SpamapS | mordred: single quotes works | 20:59 |
| jeblair | hashar: i think it may need a little more than that; i'm working on a test, should have something in a bit | 20:59 |
| fungi | YorikSar: yeah, for this time around, i'm going to settle for join queries on the database | 21:00 |
| ArxCruz | anteaya: it seems to work now :) | 21:00 |
| anteaya | ArxCruz: awesome, thank you | 21:00 |
| krtaylor | \o/ | 21:00 |
| anteaya | \o/ | 21:01 |
| jogo | I am confused at why this failed to merge: https://review.openstack.org/#/c/90266/ | 21:01 |
| *** pdmars has quit IRC | 21:01 | |
| anteaya | krtaylor ArxCruz do you feel like whipping up an email for the 3rd party ci people? I'm sure they have missed all of ours | 21:01 |
| anteaya | krtaylor ArxCruz perhaps they will listen to you | 21:02 |
| *** pdmars has joined #openstack-infra | 21:02 | |
| anteaya | and it will give me something to point to in the next few days | 21:02 |
| zaro | jogo: i believe you need verified+1 before something can merge | 21:02 |
| *** dkehn is now known as dkehn_afk | 21:02 | |
| jogo | zaro: right, and jenkins said it couldn't be merged | 21:02 |
| jogo | This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset. | 21:02 |
| hashar | that is usually Zuul not being to merge the change with the tip of the branch | 21:03 |
| hashar | but Zuul might fails for another reason though :/ Like not being to update the tip of the branch from Gerrit if it is dead | 21:03 |
| jogo | hashar: I rebased the branch locally and its at master | 21:03 |
| hashar | (at least on my setup, which is not OS one) | 21:03 |
| *** mfer has quit IRC | 21:03 | |
| anteaya | the patch comments seem to be "open" by default, I don't see a "closed" or "mark as read" type appearance | 21:03 |
| hashar | jogo: I think it is on the server side :/ | 21:04 |
| jogo | exactly why I am wondering what happened | 21:04 |
| jogo | zaro: ^ | 21:04 |
| jeblair | + success: | 21:04 |
| jeblair | + gerrit: {} | 21:04 |
| jeblair | hashar: ^ we'll have to do that unless we do something special to change the yaml syntax | 21:04 |
| ArxCruz | anteaya: done | 21:05 |
| anteaya | ArxCruz: thanks | 21:05 |
| *** sarob has joined #openstack-infra | 21:05 | |
| anteaya | ArxCruz: where did you post it to? | 21:05 |
| ArxCruz | anteaya: openstack-infra maillist | 21:06 |
| anteaya | okay great | 21:06 |
| hashar | jeblair: apparently :( since you experimental only has a message which is defined internally | 21:06 |
| *** pdmars has quit IRC | 21:06 | |
| ArxCruz | anteaya: and twitter right now :) | 21:07 |
| jogo | jeblair: any ideas on why this is failign to merge: it works locally -- https://review.openstack.org/#/c/90266/3 | 21:07 |
| jeblair | hashar: https://review.openstack.org/90820 can you take a look at that? | 21:07 |
| hashar | jeblair: sure | 21:09 |
| jeblair | jogo: yeah, more host key problems | 21:09 |
| jogo | jeblair: on my side or your side? | 21:09 |
| greghaynes | Im having a similar issue to jogo with https://review.openstack.org/#/c/90800/ | 21:09 |
| anteaya | ArxCruz: go you | 21:09 |
| jeblair | jogo: try again now | 21:09 |
| jeblair | fungi: are you done with cleaning up wip? | 21:10 |
| jogo | jeblair: no dice | 21:10 |
| YorikSar | jeblair: We can remove {} if we tolerate None in params. | 21:10 |
| jogo | https://review.openstack.org/#/c/90266/ still failing to merge | 21:10 |
| hashar | jeblair: might want to update doc as well. But I guess it can be done later | 21:10 |
| fungi | jeblair: still working on it. i have the api query list i need to run almost finished | 21:11 |
| *** matty_dubs is now known as matty_dubs|gone | 21:11 | |
| hashar | jeblair: sounds good :) | 21:12 |
| jeblair | hashar, YorikSar: yes, we could change the layoutvalidator to accept (None or dict) there... | 21:13 |
| jeblair | zuul assumes it's a dict though, so we'd need to change a few references to that... | 21:13 |
| hashar | I am trying to find the documentation for the reporters | 21:13 |
| hashar | ah http://ci.openstack.org/zuul/reporters.html#gerrit | 21:13 |
| *** gokrokve has joined #openstack-infra | 21:14 | |
| jeblair | https://etherpad.openstack.org/p/8upXBWjktL | 21:15 |
| jeblair | YorikSar, hashar, mordred, fungi: ^ | 21:15 |
| jeblair | YorikSar, hashar, mordred, fungi: preferences? | 21:15 |
| *** mrmartin has quit IRC | 21:16 | |
| *** thomasem has quit IRC | 21:16 | |
| anteaya | jeblair: so option B would work now and option A would require a patch to zuul? | 21:16 |
| fungi | jeblair: option b has less of a chance to get mis-represented by an indentation error | 21:16 |
| jeblair | jogo: try again | 21:17 |
| jeblair | anteaya: they both require a patch; option a requires a bigger patch, option b only needs https://review.openstack.org/#/c/90820/ | 21:17 |
| hashar | jeblair: is that going to change the layout ? | 21:17 |
| jogo | jeblair: :( | 21:17 |
| jogo | jeblair: still nothing | 21:18 |
| Alex_Gaynor | If we've got approved patches which didn't land and aren't in the pipeline, should we recheck no bug, or is someone going to batch do that? | 21:18 |
| jeblair | sheesh | 21:18 |
| YorikSar | jeblair: Yes. On another look opion B looks better. | 21:18 |
| ArxCruz | anteaya: in my case I had to run ssh -i /var/lib/zuul/ssh/id_rsa -p 29418 powerkvm@review.openstack.org gerrit ls-projects | 21:18 |
| jeblair | btw, openstack sucks at taking a week off. | 21:18 |
| jogo | jeblair: heh yeah | 21:18 |
| ArxCruz | perhaps is better add a note that it might need to pint the ssh key | 21:18 |
| ArxCruz | s/pint/point to | 21:18 |
| jeblair | that's my feedback for the next release cycle session. :) | 21:18 |
| *** dkliban_afk is now known as dkliban | 21:18 | |
| *** gokrokve has quit IRC | 21:19 | |
| anteaya | gerrit server is working awefully hard between page loads on that patch, for me | 21:19 |
| *** aysyd has quit IRC | 21:19 | |
| jeblair | ArxCruz: yeah, i've done that but am still seeing RSA host key for IP address '2001:4800:7815:104:3bc3:d7f6:ff03:bf5d' not in list of known hosts. | 21:19 |
| jeblair | i also did that with the ip addr | 21:19 |
| ArxCruz | jeblair: it's working for me now :) | 21:19 |
| anteaya | huh, must just be me: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=9&page=3 | 21:20 |
| hashar | YorikSar: jeblair mordred fungi : got a third option which is to explicitly ask the reporter to report a message | 21:20 |
| *** ociuhandu has quit IRC | 21:21 | |
| *** nati_ueno has quit IRC | 21:21 | |
| YorikSar | hashar: Why would you want to set message: false? | 21:21 |
| anteaya | so far {} is winning in my books | 21:22 |
| jeblair | puppet! | 21:22 |
| hashar | YorikSar: there is no point indeed. Unless you just want it to just vote | 21:22 |
| jeblair | i think known_hosts on the zuul servers is managed by puppet | 21:22 |
| mordred | jeblair: back - reading options | 21:22 |
| mordred | jeblair: {} | 21:23 |
| hashar | YorikSar: the idea is to make it more obvious when reading the layout file. gerrit: {} is not that readable | 21:23 |
| jeblair | jogo: found the problem | 21:23 |
| *** jeblair changes topic to "Known problem with WIP changes. Known problem with bogus merge failures. Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade" | 21:24 | |
| anteaya | hashar: I find {} more recongizable then no characters or true | 21:24 |
| mordred | jeblair: what can I do to be useful to you right now? | 21:24 |
| jeblair | mordred: gerritbot | 21:25 |
| mordred | jeblair: k. on it | 21:25 |
| mriedem | WIP is back, \o/ | 21:25 |
| mriedem | thanks | 21:25 |
| fungi | okay, owner-impersonated workflow -1 votes have now been successfully added to the ~300 former wip-status patchsets (except for a dozen where it looks like gerrit was unable to reverse the owner's accounts.preferred_email into an actual impersonatable account for some reason... working on those now) | 21:25 |
| jogo | jeblair: cool, thanks | 21:25 |
| YorikSar | hashar: Can there be a list of reporters? | 21:25 |
| hashar | anteaya: yup probably :] | 21:25 |
| hashar | YorikSar: I think so. There is a smtp reporter as well | 21:26 |
| anteaya | my WIP changes are back, thanks fungi | 21:26 |
| *** jooools has quit IRC | 21:26 | |
| *** imcsk8 has quit IRC | 21:26 | |
| mrodden | i think i may have found a Gerrit bug... or an unexpected behavior. | 21:26 |
| *** imcsk8 has joined #openstack-infra | 21:26 | |
| YorikSar | hashar: let me add option D then... | 21:26 |
| mrodden | this patchset https://review.openstack.org/#/c/85421/2 was merged on milestone-proposed | 21:27 |
| jeblair | mordred: https://review.openstack.org/90827 | 21:27 |
| mrodden | but this patchset was dependent ( a child of the above patchset, merged on milestone-proposed) but was merged into master https://review.openstack.org/#/c/82264/ | 21:27 |
| mordred | +2 | 21:27 |
| *** afazekas has joined #openstack-infra | 21:27 | |
| jeblair | fungi: can you force-merge https://review.openstack.org/#/c/90827/1 please? | 21:28 |
| mrodden | which brought in code to master that was originally only supposed to be for milestone-proposed | 21:28 |
| fungi | jeblair: done, and lgtm | 21:28 |
| jeblair | mrodden: agreed; would be interesting to test with gerrit 2.8 (since that happened under gerrit 2.4) | 21:29 |
| mrodden | jeblair: ok so it was a 2.4 thing | 21:29 |
| mrodden | that was my guess | 21:29 |
| mrodden | but its horrifying. | 21:30 |
| YorikSar | hashar, fungi, jeblair, mordred, anteaya: Added option D there. | 21:30 |
| jeblair | mrodden: it still may be in effect | 21:30 |
| mrodden | theoretically i could take master, put a patch on it, propose it to stable/icehouse | 21:30 |
| mrodden | and if someone merged that patch, it would look like master entirely | 21:30 |
| hashar | YorikSar: that would erase the vote made by another pipeline :D | 21:31 |
| *** aysyd has joined #openstack-infra | 21:31 | |
| jeblair | hashar: i think the main thrust of YorikSar's suggestion is the change from dict to list | 21:31 |
| YorikSar | hashar: I've added parameter just for example. | 21:31 |
| *** nati_ueno has joined #openstack-infra | 21:31 | |
| jeblair | YorikSar: that breaks backwards compat though; if we do that, i'd rather have a migration plan | 21:31 |
| *** maelfius has joined #openstack-infra | 21:32 | |
| jeblair | i'm leaning toward option B for now, it's simple and explicit, and it doesn't preclude us from doing any of the others later. | 21:32 |
| hashar | YorikSar: but yeah that was my workaround for my use case ( a pipeline that runs on merged changes. I have added verified: 0 to let zuul message, gerrit 2.8 ignore the score in such when voting on closed changes) | 21:32 |
| YorikSar | jeblair: We can just wrap dict into a list. | 21:32 |
| mrodden | i really hop that bug is fixed in 2.8 | 21:32 |
| mrodden | hope* | 21:32 |
| jeblair | fungi: can you review and force-merge https://review.openstack.org/#/c/90820/ ? | 21:33 |
| fungi | looks like the handful of impersonations which failed were due to multiple accounts with the same e-mail address | 21:33 |
| mrodden | it might be a bug with Zuul though, since that is doing the merges | 21:33 |
| *** adalbas has quit IRC | 21:33 | |
| *** morganfainberg has quit IRC | 21:33 | |
| jeblair | mrodden: no, gerrit performs the actual merges, zuul just tells it to. | 21:33 |
| mrodden | oh | 21:33 |
| mrodden | ok. | 21:33 |
| *** maelfius is now known as morganfainberg | 21:33 | |
| msabramo | I also have a problem like jogo: where merges are failing; despite successful rebase locally | 21:34 |
| msabramo | https://review.openstack.org/#/c/90827/ | 21:34 |
| jeblair | msabramo: see topic | 21:34 |
| jeblair | fungi: all done with wip? | 21:34 |
| fungi | jeblair: i need to manually untangle the dupe accounts which were preventing impersonation for a small percentage of them, but the rest are fixed | 21:35 |
| jeblair | oh ok | 21:35 |
| jeblair | jogo: try again | 21:35 |
| fungi | reviewing the zuul change now | 21:35 |
| mordred | jeblair: 2014-04-28 21:35:44,691 DEBUG irc.client: FROM SERVER: ERROR :Closing Link: review.openstack.org (Excess Flood) | 21:36 |
| mordred | jeblair: I believe at some point we were trying to reconnect too much and freenode is now unhappy with us | 21:36 |
| mordred | or something | 21:36 |
| jeblair | mordred: does it sleep between joining channels? | 21:36 |
| mordred | 0.5 seconds | 21:36 |
| jeblair | mordred: how far does it get before it's closed? | 21:36 |
| *** dkranz has quit IRC | 21:37 | |
| hashar | jeblair: gotta sleep sorry. gerrit: {} looks good to me :] | 21:37 |
| jeblair | hashar: cool, thanks for your help! | 21:38 |
| hashar | have a good post upgrade day | 21:38 |
| jeblair | mordred: ? | 21:39 |
| *** hashar has quit IRC | 21:39 | |
| mordred | trying to sort that out | 21:39 |
| jeblair | mordred: if it's not immediate, then we should probably sleep 2 seconds between joins | 21:39 |
| YorikSar | Got to sleep now. Thanks to everybody for great long-awaited upgrade! | 21:39 |
| *** afazekas has quit IRC | 21:39 | |
| jogo | jeblair: looks promising, didn't fail to merge yet | 21:40 |
| *** mat-lowery has left #openstack-infra | 21:40 | |
| jeblair | msabramo: ^ | 21:40 |
| *** jeblair changes topic to "Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade Gerritbot is offline. Known problem with formatting of test reports." | 21:41 | |
| *** jeblair changes topic to "Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade Gerritbot is offline. Known problem with experimental pipelines. Known problem with formatting of test reports." | 21:41 | |
| msabramo | jeblair: yeah mine looks good so far; at least it didn't fail right away - https://review.openstack.org/#/c/90542/ | 21:42 |
| mordred | jeblair: right now we're getting blocked from connecting due to excessive flood | 21:42 |
| *** kiall_ has joined #openstack-infra | 21:42 | |
| JayF | You guys might be able to get an exemption for the new IP. I have a buddy on freenode staff I can ping if it'd be helpful. | 21:43 |
| *** kiall_ is now known as Kiall2 | 21:43 | |
| *** luisg_ has joined #openstack-infra | 21:43 | |
| Kiall2 | Hey guys.. tarball job post tag-push just failed.. How would I retrigger that? https://jenkins01.openstack.org/job/designate-tarball/3/console | 21:44 |
| jogo | thanks jeblair | 21:44 |
| mordred | JayF: thanks - might ping you in a bit if we can't sort out rate limits directly | 21:45 |
| sdague | I would like to extend a warm congrats on upgrade well done. Now I'll just have to get used to CR & V columns being flipped :) | 21:45 |
| mordred | jeblair: just fyi - I've got gerritbot logging and bot.py edited in place on review.o.o | 21:45 |
| *** adalbas has joined #openstack-infra | 21:45 | |
| mordred | sdague: zaro is going to fix that for us | 21:46 |
| sdague | oh, cool | 21:46 |
| *** UtahDave has quit IRC | 21:46 | |
| sdague | yeh, there are a couple other css tweaks that might be worth it in the process as well | 21:46 |
| JayF | mordred: I ping'd him and asked him to /j here when/if he gets a sec | 21:47 |
| jeblair | sdague: there is so much ui to fix | 21:47 |
| jeblair | mordred: may i run the bot? | 21:48 |
| mordred | jeblair: please | 21:48 |
| mordred | jeblair: I added a debug logger for the irc stuff | 21:49 |
| sdague | jeblair: well at least on the old screen it's not total vomit :) | 21:49 |
| mordred | in /var/log/gerritbot/gerritbot_irc.log | 21:49 |
| chmouel | anteaya: cool thnaks (about the review tools session) | 21:49 |
| mordred | jeblair: and I changed the delay from 0.5 to 1.5 just to see if it would get further | 21:49 |
| jeblair | mordred: how are you editing? | 21:49 |
| mordred | jeblair: edited /etc/gerritbot/logging.yaml in place | 21:49 |
| jeblair | mordred: how did you change the time? | 21:49 |
| chmouel | anteaya: about the plugin no worries, i was just mentioning that i like the idea about this plugin :) whenever you guys would have time | 21:50 |
| mordred | jeblair: same thing- just touched the installed file in /usr/local/lib - planned on just re-running setup.py install when I was done | 21:50 |
| mordred | jeblair: /usr/local/lib/python2.7/dist-packages/gerritbot/bot.py | 21:50 |
| *** weshay has quit IRC | 21:51 | |
| jeblair | mordred: there is no sleep between joins | 21:51 |
| mordred | there isn't? | 21:51 |
| jeblair | mordred: and the latest output in the log looks like it spews all the joins at the server immediately | 21:51 |
| mordred | jeblair: heh. thank you - I didn't think to actually look at the tiemstamps :( | 21:52 |
| jeblair | mordred: edited to add sleeps | 21:52 |
| jeblair | mordred: how do you run it? | 21:52 |
| mordred | jeblair: /etc/init.d/gerritbot start | 21:52 |
| mordred | jeblair: there's probably a better way to do that | 21:53 |
| *** ihrachyshka has quit IRC | 21:53 | |
| kiall | So .. nobody knows how I can retigger a failed job triggered by a tag push? Looks like a replication race or something :) https://jenkins01.openstack.org/job/designate-tarball/3/ | 21:53 |
| mordred | that's looking better! | 21:53 |
| mordred | jeblair: where did you add the sleep? | 21:53 |
| *** smarcet has quit IRC | 21:53 | |
| jeblair | kiall: you chose an interesting time to push a tag. | 21:53 |
| msabramo | jeblair: my change finally merged and passed. Thanks for everyone who fixed this! | 21:54 |
| kiall | jeblair: really? Whoops. | 21:54 |
| *** markmcclain has quit IRC | 21:54 | |
| *** openstackgerrit has joined #openstack-infra | 21:54 | |
| *** markmcclain1 has joined #openstack-infra | 21:54 | |
| jeblair | kiall: yeah, it's openstack's week off and we're upgrading gerrit. | 21:54 |
| mordred | kiall: we're still working through issues with this morning's gerrit upgrade | 21:54 |
| kiall | Ahhh | 21:54 |
| kiall | Right. /me should have known.. didn't... heh | 21:54 |
| jeblair | kiall: one of us will have to do that manually later on; remind us when we're not putting out fires | 21:55 |
| kiall | jeblair: sure :) | 21:55 |
| kiall | thanks | 21:55 |
| fungi | OKAY! _now_ all remaining wip patches have been updated with appropriate workflow votes | 21:55 |
| mordred | fungi: woot | 21:55 |
| * fungi should go find breakfast. it's been a... busy... day | 21:56 | |
| *** zehicle has joined #openstack-infra | 21:56 | |
| jeblair | mordred: can you make that into a change for gerritbot? | 21:56 |
| mordred | jeblair: yah | 21:57 |
| sdague | arosen: what's up with your devstack changes that are in triplicate? | 21:58 |
| *** markmcclain1 has quit IRC | 21:58 | |
| *** zehicle_at_dell has quit IRC | 21:58 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove force-message from zuul layout https://review.openstack.org/90832 | 21:59 |
| jeblair | yay gerritbot | 21:59 |
| *** markmcclain has joined #openstack-infra | 21:59 | |
| mordred | jeblair: uhm. so | 22:00 |
| *** rlandy has quit IRC | 22:00 | |
| mordred | jeblair: we run gerritbot from pip | 22:01 |
| *** zehicle has quit IRC | 22:01 | |
| jeblair | mordred: yeah, we'll need to cut a new release | 22:02 |
| mordred | jeblair: yeah - we haven't done that since 2012 | 22:02 |
| anteaya | chmouel: thanks (was putting my homemade pizza in the oven) | 22:02 |
| anteaya | fungi: yay and thank you | 22:03 |
| jeblair | mordred: we can also pip install locally | 22:03 |
| jeblair | mordred: or switch to cd | 22:03 |
| mordred | jeblair: I'm fine with either - just looking at the list of changes | 22:03 |
| jeblair | i'm restarting zuul to pick up the change we just merged | 22:04 |
| *** packet has quit IRC | 22:04 | |
| mordred | jeblair: also, I _don't_ think we're running gerritbot from pip for real on gerrit | 22:04 |
| mordred | or, oh, wait - yeah | 22:04 |
| mordred | nevermind | 22:04 |
| mordred | ignore me | 22:04 |
| jeblair | /ignore mordred | 22:04 |
| *** packet has joined #openstack-infra | 22:05 | |
| mordred | jeblair: there were sleeps in both places you added sleeps- you want me to just update the old value with the new? | 22:06 |
| *** jeblair changes topic to "Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade Known problem with experimental pipelines. Known problem with formatting of test reports." | 22:06 | |
| zaro | mordred: label ordering might a little more tricky than meets the eye. i don't think it's appropriate to match column order with definition from project.config | 22:06 |
| mordred | zaro: what order do you think it should have? | 22:06 |
| jeblair | mordred: nah, don't worry about it then. | 22:06 |
| *** homeless has quit IRC | 22:07 | |
| mordred | jeblair: well, I'm still confused as to why it wasn't sleeping between joins, since there was a thing there | 22:07 |
| jeblair | mordred: because there wasn't | 22:07 |
| *** zzelle has left #openstack-infra | 22:07 | |
| jeblair | mordred: i promise i added it to the thing on disk | 22:07 |
| mordred | it was at the end of the block | 22:07 |
| mordred | OMG | 22:08 |
| mordred | WOW | 22:08 |
| mordred | what the heck ... | 22:08 |
| zaro | mordred: it's because child projects can also define labels, so when you have a collection of labels from parent+children there's many ways to set that order. | 22:08 |
| pelix | Any thoughts on adding a rules file to projects to automatically set the approved to +1 if a change receives two +2 CR and no need for core reviewers to remember once the criteria is reached? | 22:08 |
| mordred | pelix: yeah - we don't do that on purpose | 22:09 |
| JayF | I wouldn't like if that changed the default | 22:09 |
| zaro | mordred: i think the best solution would be to let the clients set the order because it's basically just the visual part of gerrit. | 22:09 |
| *** dims has quit IRC | 22:09 | |
| pelix | mordred: any particular reason why? | 22:09 |
| zaro | mordred: gerrit gwt is all magic to me right now :( | 22:10 |
| mordred | pelix: because there are many times where a second +2 is not an automatic approve | 22:10 |
| * jeblair adds "can display review columns in correct order" to gertty's feature list | 22:10 | |
| mordred | jeblair: ++ | 22:10 |
| anteaya | jeblair: will gertty take care of where the line numbers are in the diff view? | 22:10 |
| jeblair | mordred: did i mention that gertty does single-page diff? i think i forgot to put that in the readme | 22:10 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/gerritbot: Make sure we sleep a little more https://review.openstack.org/90835 | 22:10 |
| mordred | jeblair: there is your change applied to current head | 22:11 |
| mordred | jeblair: we should probably land the gertty patch | 22:11 |
| jeblair | mordred: my change shouldn't be necessary | 22:11 |
| jeblair | mordred: there was no sleep in the join loop before | 22:11 |
| jeblair | so whatever was installed must not have been current head | 22:12 |
| mattoliverau | Morning all, how'd the upgrade go? | 22:12 |
| *** hdd has quit IRC | 22:12 | |
| anteaya | mattoliverau: so far so good | 22:12 |
| anteaya | mattoliverau: still doing clean-up | 22:12 |
| anteaya | the change in the gui is irksome to some | 22:12 |
| anteaya | but that will get smoothed out in the next while | 22:13 |
| mattoliverau | anteaya: it was a big jump, but worth it, nice work everyone! | 22:13 |
| jeblair | anteaya: gertty's diff view has always been line numbers on left. ;) | 22:13 |
| anteaya | jeblair: ah, yay gertty | 22:13 |
| mattoliverau | I assume there will be more features to work with in the future, and still the best move forward. | 22:13 |
| anteaya | jeblair: have ween been using gertty and I didn't know | 22:13 |
| fungi | oh, also, i was able to finally get a nodepool image to rebuild in rackspace with the current fixes, nova boot an instance from it and reach it over the network | 22:13 |
| anteaya | the way to talk about it makes me feel like I should already know it | 22:14 |
| jeblair | fungi: !!! | 22:14 |
| anteaya | fungi: yay | 22:14 |
| anteaya | mattoliverau: /nod | 22:14 |
| *** msabramo has quit IRC | 22:14 | |
| mordred | fungi: w00t | 22:14 |
| *** gokrokve has joined #openstack-infra | 22:14 | |
| mattoliverau | I get 503 as well | 22:15 |
| mattoliverau | sorry wrong channel :P | 22:15 |
| mordred | jeblair: it's got 2 +2's - I can aprv it if you like | 22:15 |
| *** dizquierdo has joined #openstack-infra | 22:15 | |
| jeblair | mordred: so i'm assuming gerritbot was running something close to master on the old server, does that sound rightL | 22:16 |
| jeblair | ? | 22:16 |
| jeblair | mordred: (and it's just the tagged release on pip that lacks the sleeps?) | 22:16 |
| * anteaya eats pizza while it is hot | 22:16 | |
| mordred | jeblair: yeah | 22:16 |
| jeblair | mordred: so we can land the increase sleep change if you want, but it's not necessary -- the necessary thing is either to tag or switch to cd | 22:17 |
| morganfainberg | mordred, jeblair , i think i might have found an issue w/ gerrit config, +1 Workflow is "approve" right? | 22:17 |
| mordred | jeblair: ya. with you | 22:17 |
| *** msabramo has joined #openstack-infra | 22:17 | |
| mordred | morganfainberg: that's right | 22:18 |
| *** flaper87 is now known as flaper87|afk | 22:18 | |
| morganfainberg | i'm seeing (new dashboard) the ability to +1 Workflow on devstack-gate | 22:18 |
| morganfainberg | i'm not core. | 22:18 |
| morganfainberg | not just -1 workflow | 22:18 |
| morganfainberg | -1/0 | 22:18 |
| morganfainberg | i can't +2 CR, but it looks like i can +1W | 22:18 |
| *** gokrokve has quit IRC | 22:19 | |
| mordred | you can -1/0/+1 on devstack-gate? | 22:19 |
| morganfainberg | yep | 22:19 |
| morganfainberg | well the UI says so | 22:19 |
| morganfainberg | let me try in a sec | 22:19 |
| jeblair | morganfainberg: your own change or someone else's? | 22:19 |
| mordred | morganfainberg: change number? | 22:19 |
| morganfainberg | jeblair, my own change | 22:19 |
| morganfainberg | mordred, https://review.openstack.org/#/c/90812/2 | 22:19 |
| *** mwagner_lap has joined #openstack-infra | 22:19 | |
| morganfainberg | ahy | 22:20 |
| morganfainberg | it's a UX bug | 22:20 |
| morganfainberg | it doens;'t let me | 22:20 |
| morganfainberg | it puts up a really big error saying "+1 Workflow restricted" | 22:20 |
| mordred | nice | 22:20 |
| morganfainberg | yeah. | 22:20 |
| morganfainberg | meh, poor UX design, but thats what i get for using the "new" screen | 22:20 |
| jeblair | zaro: can you look into that please ^ | 22:20 |
| *** sarob has quit IRC | 22:20 | |
| jeblair | morganfainberg: oh, it's only on the new screen? | 22:20 |
| *** sarob has joined #openstack-infra | 22:21 | |
| mordred | might be the way they draw the selector box things | 22:21 |
| zaro | i don't think you can change back to old screen | 22:21 |
| morganfainberg | jeblair, yeah just noticed it wasn't on the old screen when i flipped back | 22:21 |
| morganfainberg | jeblair, sorry for raising the flag. | 22:21 |
| morganfainberg | zaro, you can, it's in perferences | 22:21 |
| jeblair | zaro: the old screen is the default and you can definitely change back and forth | 22:21 |
| morganfainberg | zaro, server default is "old screen" | 22:21 |
| mordred | morganfainberg: no - that's good - it's good to know | 22:21 |
| morganfainberg | :) | 22:22 |
| jeblair | one more reason to be terrified of the new change screen | 22:22 |
| mordred | jeblair: I think we should tag gerritbot | 22:22 |
| mordred | if we want to move to CD later, cool | 22:22 |
| morganfainberg | jeblair, i actually like some of the changes in the new screen. but..... | 22:22 |
| sdague | morganfainberg: yeh, the new screen is kind of a mess | 22:22 |
| mordred | jeblair: you want me to do it/ | 22:22 |
| mordred | ? | 22:23 |
| zaro | new screen breakdown: http://etherpad.wikimedia.org/p/new-gerrit-change-view-comments | 22:23 |
| *** nati_uen_ has joined #openstack-infra | 22:23 | |
| jeblair | mordred: just did it | 22:23 |
| zaro | morganfainberg: ^ | 22:23 |
| morganfainberg | zaro, ah nice. will add my feedback | 22:23 |
| jeblair | the new screen has some serious ui problems | 22:23 |
| morganfainberg | jeblair, the worst was "reply" = "review" | 22:24 |
| morganfainberg | took me clicking around a bunch to figure out that was how to score things | 22:24 |
| mordred | jeblair: aweomse | 22:24 |
| *** etoews has quit IRC | 22:25 | |
| *** sarob_ has joined #openstack-infra | 22:25 | |
| mordred | basically, the UI is the thing that gerrit has been historically the worst at - and in the new UI, it has gotten much better at being the worst at UI | 22:25 |
| *** sarob has quit IRC | 22:26 | |
| *** reed has quit IRC | 22:26 | |
| morganfainberg | mordred, ++ | 22:26 |
| mordred | which is a shame - because the engine does some pretty amazing things | 22:26 |
| *** nati_ueno has quit IRC | 22:26 | |
| jeblair | the whole screen is too wide (for users who don't have super-wide displays), and the comment lines are too long (long lines hinder reading), and then the information architecture is mostly about how much can be crammed into random places | 22:26 |
| jeblair | anyway, this is distracting... i'll save my new screen ranting for some other time | 22:26 |
| mordred | :) | 22:27 |
| *** jgrimm has quit IRC | 22:27 | |
| mordred | jeblair: I think we're good to re-enable puppet on gerrit | 22:27 |
| *** ociuhandu has joined #openstack-infra | 22:27 | |
| jeblair | mordred: let me check what i have in my env on puppetmaster | 22:27 |
| *** jdurgin has quit IRC | 22:27 | |
| mordred | jeblair: (I put patches for that up a little while ago which I think have landed) | 22:28 |
| mordred | unless you've added new thingses | 22:28 |
| mattoliverau | Took me a while to figure out how to make a code review (vote) on the new view as well. | 22:28 |
| anteaya | should I still be following: https://etherpad.openstack.org/p/J8WNqyP2dC or have we moved on? | 22:29 |
| sdague | mattoliverau: there is a reason the old view is the default :) | 22:29 |
| *** dizquierdo has quit IRC | 22:29 | |
| morganfainberg | sdague, ++ | 22:29 |
| jeblair | mordred: i don't see the mysql changes in config HEAD | 22:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Plumb mysql host through gerrit.pp https://review.openstack.org/90837 | 22:30 |
| jeblair | mordred: we need to land that ^ | 22:30 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/jeepyb: Use configured mysql host https://review.openstack.org/90838 | 22:31 |
| jeblair | mordred: and that ^ | 22:32 |
| mordred | both are +2 from me | 22:32 |
| jeblair | mordred: also look at https://review.openstack.org/90832 | 22:33 |
| zaro | mordred: what happened? i reviewed 90837 earlier but it looks completly new | 22:33 |
| mordred | +2 | 22:34 |
| anteaya | yeah I am looking at the same: https://review.openstack.org/#/c/90807/1/modules/openstack_project/manifests/gerrit.pp | 22:34 |
| zaro | opps i meant 90837 | 22:34 |
| mordred | zaro: no - you reviewed the version I tossed up - look at jim's patch | 22:34 |
| anteaya | mordred: what is the difference between jims and yours | 22:34 |
| anteaya | besides author | 22:35 |
| mordred | anteaya: jim wrote his - and mine - so it makes mre sense to land jim's | 22:35 |
| *** doug-fish has left #openstack-infra | 22:35 | |
| *** sarob_ has quit IRC | 22:35 | |
| anteaya | oh okay, I was looking for a difference in the code | 22:35 |
| anteaya | what about zaro's question on 90837? | 22:36 |
| *** dims has joined #openstack-infra | 22:37 | |
| anteaya | 90807 | 22:37 |
| mordred | oh - wow. workflow WIP shows as a big red X in the UI | 22:37 |
| mordred | that's gonna take some getting used to | 22:37 |
| *** dkehn_afk is now known as dkehn | 22:37 | |
| anteaya | yeah | 22:38 |
| anteaya | and abandoned does the same, sometimes | 22:38 |
| mordred | zaro, anteaya: I think we don't need to do that in this patch - the gerrit module has it as a default - but none of ours do | 22:38 |
| mordred | which is, I think, correct | 22:38 |
| anteaya | I haven't figured out when abandoned does show up as a big red x | 22:38 |
| jeblair | mordred: that's the downside of gerrit's answer to our ui changes always being "but you can make your own searches and bookmark them". the main program's ui never gets better. | 22:39 |
| anteaya | okay, I just didn't want to gloss over khai's question | 22:39 |
| mordred | anteaya: ++ | 22:39 |
| mordred | jeblair: yah | 22:39 |
| mordred | jeblair: this is the reason I can't wait to get a copy of gertty - so I can stop using the main program's UI | 22:39 |
| jeblair | i can't seem to push the tag for gerritbot | 22:40 |
| anteaya | I'll +1 jim's patch and zaro can do what he likes | 22:40 |
| mrodden | i find it sad that the direction they are going with the Gerrit UI isn't like the original "spartan" clean and simple design | 22:40 |
| mrodden | its such a great tool otherwise | 22:40 |
| anteaya | zaro: I am not disagreeing with you, just saying it is up to you to decide | 22:40 |
| mordred | jeblair: I think that there isa new "pushSignedTag" thing in gerrit now, isn't there? | 22:41 |
| mordred | jeblair: let me try adding that to the config so you can try again | 22:41 |
| anteaya | mrodden: maybe java devs don't naturally consider ux? I don't know is that an incorrect sweeping generalization? | 22:41 |
| *** msabramo1 has joined #openstack-infra | 22:41 | |
| jeblair | kiall: when you pushed a tag earlier, was it signed? | 22:41 |
| *** jdurgin has joined #openstack-infra | 22:42 | |
| mordred | oh! neat! | 22:42 |
| mordred | jeblair: https://review.openstack.org/#/c/90839/ | 22:42 |
| mrodden | anteaya: after looking at the new UI mode in 2.9, i think they are going after some kind of "social coding" UI idea | 22:42 |
| mordred | acl changes can produce review snow :) | 22:42 |
| mrodden | it has avatars and crap... | 22:42 |
| kiall | jeblair: my first attempt to push a signed tag was rejected, switched to an annotated tag after.. | 22:42 |
| mattoliverau | anteaya: I'm a java dev in a past life... and nope it's not an incorrect sweeping generalization :P | 22:42 |
| jeblair | kiall: nice. we're not going to publish that. you should only ever push signed tags. | 22:42 |
| mrodden | lol or that... | 22:42 |
| mordred | jeblair: well, the review is neat - but I have power to approve it | 22:43 |
| kiall | jeblair: hah .. I thought so, but.. Yea, it didn't click. | 22:43 |
| mordred | if you're ok with me trying that now, I'll just do it old school | 22:43 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/devstack-gate: Allow setting services up under apache mod_wsgi https://review.openstack.org/90812 | 22:43 |
| jeblair | mordred: yeah, then we'll batch change the whole system once we verify it. | 22:43 |
| mordred | jeblair: done. try pushing tag again | 22:44 |
| jeblair | * [new tag] 0.2.0 -> 0.2.0 | 22:44 |
| *** msabramo has quit IRC | 22:44 | |
| *** aconrad has quit IRC | 22:44 | |
| jeblair | mordred: want to whip up a batch acl change? | 22:44 |
| mordred | jeblair: oh do I ever | 22:44 |
| zaro | anteaya: works either way. imo one way seems more correct than the other. | 22:44 |
| mordred | jeblair: I wonder if you need pushTag as well | 22:44 |
| *** pelix has quit IRC | 22:44 | |
| mordred | kiall: wanna be a guinea pig for something/ | 22:45 |
| mordred | ? | 22:45 |
| jeblair | mordred: how about the sandbox? | 22:45 |
| zaro | anteaya: 'x' is for the largest negative vote. | 22:45 |
| kiall | ^ sounds like a better idea :P | 22:45 |
| mordred | jeblair: yah. ok | 22:45 |
| fungi | oh, wow. gerrit decided to add a new acl to limit who could push signed tags, and made the old acl only cover allowing annotated tags? that's so completely backwards | 22:45 |
| jeblair | fungi: :) | 22:46 |
| mordred | jeblair: btw - it doesn't look like the refs/tags/* entries are needed at all | 22:46 |
| anteaya | mordred: so I can comment on https://review.openstack.org/#/c/90839/1 but I have no vote | 22:46 |
| mordred | anteaya: same here. I'm just goign to abandon that | 22:46 |
| anteaya | mrodden: ewww, why do i need a picutre of you, you are right here | 22:46 |
| anteaya | mordred: but why did that patch do that? | 22:46 |
| anteaya | mattoliverau: ah okay, didn't want to hurt feelings | 22:47 |
| jeblair | mordred: https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 22:47 |
| zaro | anteaya: that's a project config change. i think only admins have voting rights there | 22:47 |
| jeblair | mordred: i added push signed tag only | 22:48 |
| anteaya | zaro: hmmmm, okay | 22:48 |
| anteaya | learning new things, thanks | 22:48 |
| sdague | zaro: if I wanted to try to submit some gerrit css fixes, which branch should that be on? | 22:48 |
| zaro | sdague: openstack/2.8.4 | 22:49 |
| sdague | ok, cool | 22:49 |
| jeblair | mordred: wrong reference; updated | 22:49 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Move other checks into their own file https://review.openstack.org/90266 | 22:49 |
| *** jhesketh_ has joined #openstack-infra | 22:49 | |
| *** jhesketh_ is now known as jhesketh | 22:50 | |
| jeblair | mordred: the current config of sandbox allows signed tags and disallows unsigned tags according to my testing | 22:50 |
| mordred | jeblair: awesome | 22:50 |
| sdague | zaro: which css file should I be proposing fixes into? | 22:50 |
| mordred | jeblair: mind if I try something else real quick? | 22:50 |
| fungi | sdague: zaro: we want css changes to be external to the gerrit source, and only apply things which are already or nearly accepted upstream to the gerrit fork we've got, right? it would be nice for the openstack-infra/gerrit project to disappear soonish | 22:50 |
| jeblair | mordred: ++ | 22:50 |
| jhesketh | Morning | 22:50 |
| sdague | fungi: sure, just point me to where that should be | 22:51 |
| jeblair | fungi, sdague, zaro: gerrit's css is meant to be customized locally | 22:51 |
| jeblair | fungi, sdague, zaro: and we have customized versions, i think that's fine | 22:51 |
| *** prad_ has quit IRC | 22:51 | |
| jeblair | fungi, sdague, zaro: having said that -- making upstream look nicer would be nice (we have done that once before already -- we are the reason why it isn't lime-green by default anymore) | 22:52 |
| fungi | jeblair: sdague: zaro: should we start forking and installing the css via openstack-infra/config instead? | 22:52 |
| *** msabramo1 has quit IRC | 22:52 | |
| zaro | jeblair: where are those local customizations located? | 22:52 |
| anteaya | jhesketh: morning | 22:52 |
| jeblair | fungi: it's not forking, it's customizing | 22:52 |
| sdague | fungi: it looks like it's compiling and compressing all the css into 1 file | 22:52 |
| *** Kiall2 has quit IRC | 22:52 | |
| jeblair | sdague: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/gerrit | 22:53 |
| jeblair | fungi: ^ | 22:53 |
| jeblair | sdague, fungi: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/gerrit/GerritSite.css | 22:53 |
| fungi | ahh, yep, so we're already doing that--all the better | 22:54 |
| sdague | jeblair: where do you think that's getting picked up? | 22:54 |
| jeblair | sdague: that file is a config option for gerrit | 22:55 |
| sdague | ah... ok | 22:55 |
| sdague | then it compiles it in | 22:55 |
| sdague | ? | 22:55 |
| jeblair | sdague: wouldn't surprise me, but i can't speak to that. | 22:55 |
| jeblair | sdague: it is a run-time config option though | 22:55 |
| *** mriedem has quit IRC | 22:56 | |
| sdague | yeh "compile" is loosely defined here | 22:56 |
| jeblair | so we're talking about compile-on-startup not compile-at-compile-time :) | 22:56 |
| * jhesketh looks like he's missed all the fun | 22:56 | |
| *** aconrad has joined #openstack-infra | 22:56 | |
| jeblair | hardly; fun continues | 22:56 |
| jhesketh | darn, I was about to go back to bed | 22:56 |
| sdague | yeh, it's being just dumped straight inline actually | 22:56 |
| sdague | ok, good | 22:56 |
| *** sarob has joined #openstack-infra | 22:56 | |
| *** mbacchi has quit IRC | 22:57 | |
| *** Sukhdev has quit IRC | 22:58 | |
| *** prad has joined #openstack-infra | 22:58 | |
| jeblair | if self.waitForRefSha(change.project, ref, change._ref_sha): | 23:00 |
| jeblair | that's taking a long time in zuul | 23:00 |
| jeblair | that's why it seems to be sitting there for a while | 23:00 |
| jeblair | that whole idea is still problematic and we need to figure out a way to get rid of it | 23:01 |
| *** etoews has joined #openstack-infra | 23:01 | |
| jeblair | though i suspect at the moment we've missed something in the move | 23:01 |
| *** aconrad has quit IRC | 23:02 | |
| jeblair | yeah, that's always failing, which means the dependent queue is worst case | 23:02 |
| jeblair | not to mention, it's very slow | 23:03 |
| jeblair | dansmith: that's what you saw earlier | 23:03 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Update acls to enforce pushing signed tags https://review.openstack.org/90842 | 23:03 |
| mordred | jeblair: ^^ there ya go | 23:03 |
| jeblair | i'm going to dig into that for a few mins; biab | 23:03 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Remove H901,H902 since it is a duplicate of H714,H713 https://review.openstack.org/90269 | 23:04 |
| jerryz_ | fungi: could you please take a look at this tiny patch, https://review.openstack.org/#/c/87418/ Thanks! | 23:06 |
| mordred | jeblair, sdague, fungi: we could put an entry into the html header which loads a CSS file from a url - and we could manage that without changing gerrit | 23:07 |
| sdague | sure | 23:07 |
| *** alexpilotti has quit IRC | 23:07 | |
| sdague | though that css file is just straight dumped in | 23:07 |
| sdague | so we sort of get the same results now | 23:07 |
| sdague | with one less http fetch | 23:08 |
| jeblair | mordred: this is the last time i'm going to say this | 23:08 |
| mordred | sure - but i the problem was that we can't change it without restarting gerrit | 23:08 |
| jeblair | mordred: we do not edit the css in the gerrit source tree | 23:08 |
| jeblair | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/gerrit/GerritSite.css | 23:08 |
| mordred | jeblair: yes. I know that | 23:08 |
| jeblair | is a file that is included | 23:08 |
| jeblair | gerrit does not need to be changed | 23:08 |
| mordred | yes. I did not say it did | 23:08 |
| mordred | and I know that we're doing that | 23:09 |
| jeblair | mordred: then why did you say "and we could manage that without changing gerrit" ? | 23:09 |
| mordred | s/changing gerrit/restarting gerrit/ | 23:09 |
| jeblair | mordred: that is an very important distinction | 23:09 |
| *** mbacchi has joined #openstack-infra | 23:09 | |
| mordred | jeblair: yes it is -sorry for the confusion | 23:10 |
| mordred | I was responding to "jeblair | "so we're talking about compile-on-startup not compile-at-compile-time" | 23:10 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/devstack-gate: Allow setting services up under apache mod_wsgi https://review.openstack.org/90812 | 23:10 |
| *** bauzas has quit IRC | 23:11 | |
| *** mriedem has joined #openstack-infra | 23:11 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Check for new files as well as changed files https://review.openstack.org/90843 | 23:11 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: limit the width of vote columns https://review.openstack.org/90844 | 23:12 |
| *** gokrokve has joined #openstack-infra | 23:14 | |
| *** afazekas has joined #openstack-infra | 23:15 | |
| *** jhesketh__ has joined #openstack-infra | 23:15 | |
| *** jhesketh has quit IRC | 23:15 | |
| sdague | do I remember that we might be able to get full diffs in change email? | 23:16 |
| jeblair | sdague: we still have several regressions, can we defer discussion a bit until we have those wrapped up? | 23:17 |
| *** jhesketh__ is now known as jhesketh | 23:17 | |
| jeblair | mordred: i'm having trouble figuring out why zuul isn't dealing with them correctly | 23:18 |
| *** packet has quit IRC | 23:18 | |
| jeblair | mordred: with the ref sha thing | 23:18 |
| mordred | jeblair: if I wanted to help you look - what woudl be a good first place? | 23:18 |
| *** rcarrill` has joined #openstack-infra | 23:19 | |
| jeblair | jhesketh: any thoughts? | 23:19 |
| *** gokrokve has quit IRC | 23:19 | |
| mordred | just the code? | 23:19 |
| jeblair | 2014-04-28 23:09:50,833 DEBUG zuul.Gerrit: Change <Change 0x7fc91439ac50 90269,3> did not appear in the git repo | 23:19 |
| jeblair | 2014-04-28 23:09:52,005 DEBUG zuul.Gerrit: Waiting for <Change 0x7fc914d99510 89390,2> to appear in git repo | 23:19 |
| jeblair | oops sorry | 23:19 |
| mordred | jeblair: are they there? | 23:19 |
| jeblair | 2014-04-28 23:04:48,554 DEBUG zuul.Gerrit: Waiting for <Change 0x7fc91439ac50 90269,3> to appear in git repo | 23:19 |
| jeblair | 2014-04-28 23:09:50,833 DEBUG zuul.Gerrit: Change <Change 0x7fc91439ac50 90269,3> did not appear in the git repo | 23:19 |
| jeblair | those are the relevant log entries | 23:19 |
| jeblair | basically, every change it has merged since the upgrade exhibits that behavior | 23:19 |
| jeblair | mordred: yes | 23:20 |
| mordred | jeblair: is the hex number the internal zuul ide? | 23:20 |
| mordred | id? | 23:20 |
| jeblair | mordred: yes | 23:20 |
| jeblair | mordred: and afaict, the replica is correct | 23:20 |
| jhesketh | jeblair: sorry, my connection dropped out.. which were you asking my thoughts on? | 23:20 |
| *** rcarrillocruz has quit IRC | 23:20 | |
| sdague | jeblair: sure | 23:21 |
| jeblair | mordred: it's a pretty dumb system actually -- it just checks to make sure the sha for the ref is different after the change merges | 23:21 |
| *** asettle is now known as asettle-bfast | 23:21 | |
| jeblair | (doesn't actually care what it is, just as long as it's not the old value | 23:21 |
| sdague | I need to make dinner anyway, I'll circle tomorrow | 23:21 |
| jeblair | sdague: thx | 23:21 |
| *** tjones has joined #openstack-infra | 23:21 | |
| mordred | jeblair: is it possible that gerrit is reporting back a different payload? | 23:21 |
| mordred | jeblair: and therefore we're seeing the orig sha and not the sha of the merge commit? | 23:21 |
| *** hogepodge has quit IRC | 23:21 | |
| jeblair | mordred: it uses the apache mirror | 23:21 |
| anteaya | jhesketh: new gerrit and zuul are not speaking well | 23:22 |
| jeblair | mordred: that's why it odesn't care | 23:22 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add noqa everywhere that it is easy to do so https://review.openstack.org/90270 | 23:22 |
| tjones | hi - i have made changes to the branch and am trying to push them upsteam and am getting remote: (W) No changes between prior commit 25607b5 and new commit 079bf04. anyone else have trouble with git review? | 23:22 |
| mordred | gotcha | 23:22 |
| anteaya | tjones: well we just upgraded gerrit | 23:22 |
| anteaya | tjones: and we are still sorting things out | 23:22 |
| *** hemna is now known as hemna__ | 23:22 | |
| *** david-lyle has quit IRC | 23:22 | |
| anteaya | tjones: so you have local changes and you are having difficulty getting them onto gerrit? | 23:22 |
| tjones | anteaya: yeah i know - just checking if it's me or something else. i'll wait patiently :-) | 23:22 |
| anteaya | tjones: can you paste your error output, I'll take a look | 23:23 |
| *** melwitt has quit IRC | 23:24 | |
| tjones | anteaya: thanks http://paste.openstack.org/show/77549/ | 23:24 |
| mordred | jeblair: project.name is coming from zuul data model right? there's no chance gerrit is sending a url escaped / ? | 23:24 |
| anteaya | mordred: I am reviewing 90842 the acl change and I have questions, but they can wait until gerrit and zuul are getting along better | 23:24 |
| jeblair | mordred: i think the apache mirror may be behind | 23:24 |
| mordred | jeblair: oh! | 23:25 |
| anteaya | tjones: what does git diff show you? | 23:26 |
| tjones | nothing at all | 23:26 |
| tjones | hmmmmmm | 23:26 |
| mordred | jeblair: icationQueue : Rescheduling replication to cgit@git05.openstack.org:/var/lib/git/openstack-dev/hacking.git to avoid collision with an in-flight push. | 23:26 |
| jeblair | mordred: yeah, the apache mirror is out of date | 23:26 |
| mordred | jeblair: there's a BUNCH of that ^^ in the log | 23:27 |
| tjones | i'l paste another. i made a change to spacing, git diff saw that, i added, committed, and review says i changed nothing | 23:27 |
| mordred | oh. hahahaha. also - not related to this - launchpadlib is not on the server, so the change hooks are unhappy | 23:27 |
| anteaya | tjones: what kind of change to spacing? | 23:28 |
| anteaya | if I remember correctly there was a patch somewhere to compress more than one whitespace to one whitespace | 23:28 |
| anteaya | clarkb was working on that | 23:28 |
| mordred | jeblair: something is WILDLY Wrong with replication | 23:28 |
| tjones | anteaya: i removed a blank line | 23:28 |
| tjones | http://paste.openstack.org/show/77550/ | 23:28 |
| anteaya | I have forgotten context though | 23:29 |
| anteaya | tjones: at the end of a file? | 23:29 |
| mordred | I've been scrolling up in the rror log file with y finger holding down the PAGE UP button for quite a while | 23:29 |
| tjones | no - midway | 23:29 |
| *** asettle has joined #openstack-infra | 23:29 | |
| mordred | and it's all just lines of that line above | 23:29 |
| tjones | it didn't add much value | 23:29 |
| *** lakshmi has quit IRC | 23:29 | |
| tjones | i can change something like ordering of init variables to see if it is whitespace dependent | 23:29 |
| anteaya | tjones: try that | 23:30 |
| anteaya | tjones: what editor are you using? | 23:30 |
| tjones | pycharm | 23:31 |
| jeblair | $ grep Rescheduling error_log|wc -l | 23:31 |
| jeblair | 2931015 | 23:31 |
| anteaya | I don't recognize | 23:31 |
| anteaya | oh | 23:31 |
| mordred | jeblair: yeah | 23:31 |
| tjones | anteaya: same issue http://paste.openstack.org/show/77551/ | 23:32 |
| mordred | jeblair: right before the first one is an error about github hostkey | 23:32 |
| tjones | see the bottom | 23:32 |
| jeblair | mordred: did we restart since then? | 23:32 |
| mordred | jeblair: I fix the github hostkey and restarted - perhaps I did not do it fully properly? | 23:32 |
| jeblair | Scheduling a retry to avoid collision with an in-flight push is differentiated from a retry due to a transport error. In the case of collision avoidance, the job is rescheduled according to the replication delay, rather than the retry delay. | 23:33 |
| *** saper_ is now known as saper | 23:33 | |
| jeblair | mordred: so 1 thing -- i think the reason we get so many is the replication delay is 0, right? | 23:33 |
| mordred | yes. that's right | 23:33 |
| tjones | anteaya: this may be a hint. I made more significant changes originally and when it said "do you really want to do this" i said no and then changed something else and then add/commit/review again and this time said yes | 23:33 |
| jeblair | so it's doing that in a tight loop | 23:33 |
| mordred | quite impressively | 23:33 |
| *** jaypipes has quit IRC | 23:34 | |
| mordred | jeblair: is it possible that new gerrit is better at something which is causing it to trip this? | 23:34 |
| mordred | jeblair: and since we do have waitForRef in zuul - something like replicationDelay=1 shouldn't be a functinoal issue, right? | 23:34 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update docs and All-Project configuration for gerrit 2.8 https://review.openstack.org/70818 | 23:35 |
| jeblair | mordred: yeah, i wonder how short we can make that? | 23:35 |
| jeblair | mordred: could we do 0.1? | 23:35 |
| *** praneshp has quit IRC | 23:35 | |
| jeblair | i'm trying to find the docs for it | 23:35 |
| anteaya | tjones: can you submit a test patch to http://git.openstack.org/cgit/openstack-dev/sandbox/ | 23:35 |
| anteaya | tjones: create a new patch with no dependencies | 23:36 |
| tjones | anteaya: sure | 23:36 |
| tjones | just a sec to set up | 23:36 |
| anteaya | sure | 23:36 |
| *** jhesketh has quit IRC | 23:37 | |
| tjones | anteaya: yes that worked fine | 23:38 |
| *** jhesketh__ has joined #openstack-infra | 23:38 | |
| tjones | https://review.openstack.org/#/c/90847/ | 23:38 |
| *** jhesketh__ is now known as jhesketh | 23:39 | |
| anteaya | tjones: okay so that indicates to me that your gerrit account has the correct permissions | 23:39 |
| tjones | anteaya: do you want me to try to make another patch depenent on this one in the sandbox? | 23:40 |
| anteaya | let's try that | 23:40 |
| jeblair | mordred: i think it's an int | 23:40 |
| mordred | jeblair: it's an int | 23:40 |
| jeblair | mordred: let's set replicationDelay to 1 for our sanity | 23:40 |
| mordred | just foudn it in the code | 23:40 |
| mordred | kk | 23:40 |
| jeblair | mordred: you want to do that and restart? | 23:40 |
| mordred | yup | 23:41 |
| *** jeblair changes topic to "Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade Known problem with replication. Known problem with experimental pipelines. Known problem with formatting of test reports." | 23:41 | |
| *** wenlock has quit IRC | 23:41 | |
| *** jeblair changes topic to "Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade Known problems with replication, experimental pipelines, and formatting of test reports." | 23:41 | |
| tjones | anteaya: bummer - it worked | 23:41 |
| tjones | https://review.openstack.org/90848 | 23:42 |
| mordred | jeblair: restarting | 23:42 |
| anteaya | tjones: okay now create another dependency and answer no to the first prompt | 23:43 |
| mordred | jeblair: restated | 23:43 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Increase replication delay to 1 second https://review.openstack.org/90850 | 23:44 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Fix py33, make sure skip_on_py3 is part of every check https://review.openstack.org/90271 | 23:44 |
| *** mnaser has left #openstack-infra | 23:44 | |
| *** smarcet has joined #openstack-infra | 23:44 | |
| * zaro needs to catch a bus home, bbl | 23:45 | |
| tjones | anteaya: still worked. i did the same steps exactly - including a rebase | 23:45 |
| tjones | https://review.openstack.org/90851 | 23:45 |
| *** etoews has quit IRC | 23:45 | |
| *** miqui has quit IRC | 23:46 | |
| *** asettle has quit IRC | 23:46 | |
| anteaya | tjones: hmmmmm | 23:46 |
| anteaya | well we have ruled out problems with your gerrit account | 23:47 |
| anteaya | now we have to look more closely at the git log of the repo you want to offer patches to | 23:47 |
| mordred | jeblair: it doesn't seem to be particularly happy about the lack of All-Projects.git on the targets | 23:47 |
| jeblair | mordred: i think that's always been an ignorable error | 23:48 |
| *** etoews has joined #openstack-infra | 23:48 | |
| *** jamielennox|away is now known as jamielennox | 23:49 | |
| mordred | wow. the documentation has gotten worse | 23:49 |
| mordred | how do I find docs on replication now? | 23:49 |
| openstackgerrit | A change was merged to openstack/requirements: troveclient 1.0.4 released https://review.openstack.org/86542 | 23:49 |
| anteaya | tjones: so this is the patch we are discussing? https://review.openstack.org/#/c/83207/ | 23:49 |
| jeblair | mordred: i never found them | 23:49 |
| jeblair | zaro: ? | 23:49 |
| jeblair | oh he's gone | 23:50 |
| jeblair | mordred: mind if i apt-get install python-launchpadlib | 23:50 |
| mordred | jeblair: please do | 23:50 |
| mordred | jeblair: I'll make a puppet change | 23:50 |
| tjones | anteaya: it is actyally the top of the dependency chaing https://review.openstack.org/#/c/86443/21 | 23:51 |
| tjones | s/chaing/chain | 23:51 |
| anteaya | tjones: you are trying to make a change to shawn hartsocks patch? | 23:51 |
| tjones | anteaya: yes i have taken them over from him | 23:51 |
| anteaya | this might be a case of patch ownership, I don't know | 23:52 |
| mattoliverau | tjones anteaya: I wonder if the errors we are seeing (reported by Jim) is the cause, they seem to signify patches are failing to land on some repos, and the continuing resheduling of replicaitons based on collisions... I wonder if tjones repo is effected by this problem. The patch fails to land, so git-review does a diff with nothing, seeing no change, etc. Maybe it's worth waiting till jim and | 23:52 |
| mattoliverau | mordred can solve the current issues. | 23:52 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add launchpadlib to review.o.o https://review.openstack.org/90852 | 23:52 |
| *** signed8bit has quit IRC | 23:53 | |
| tjones | mattoliverau: sure - if you guys can ping me when done i can resubmit the review to see | 23:53 |
| anteaya | mattoliverau: they certainly might be overlapping issues, that is true | 23:53 |
| anteaya | tjones: can you check back tomorrow? | 23:53 |
| anteaya | this is feeling like a stubborn issue and I'm sense we are all pretty tired | 23:54 |
| jeblair | mordred: do we have a hook that does something with 'gerrit gsql'? | 23:54 |
| *** praneshp has joined #openstack-infra | 23:54 | |
| mordred | jeblair: yes | 23:54 |
| mattoliverau | tjones: it's only the start of my day, I'll ping you once it's resoved :) | 23:54 |
| *** etoews has quit IRC | 23:54 | |
| anteaya | mattoliverau: thanks | 23:54 |
| mordred | or, I thought we did | 23:54 |
| jeblair | mordred: i mean, we're getting errors related to that in the logs | 23:54 |
| tjones | anteaya: sure. ok thanks mattoliverau. otherwise if i do not hear from you I will try again tomorrow am | 23:54 |
| mordred | jeblair: jeepyb trivial_rebase | 23:54 |
| anteaya | tjones: what is am for you? | 23:54 |
| *** lcostantino has quit IRC | 23:54 | |
| jeblair | [2014-04-28 23:54:26,365] ERROR com.google.gerrit.server.contact.EncryptedContactStore : Cannot store encrypted contact information | 23:54 |
| jeblair | java.io.IOException: Connection failed: 404 | 23:54 |
| tjones | anteaya: im in PDT | 23:54 |
| jeblair | that's delightful | 23:55 |
| anteaya | so 1600 utc | 23:55 |
| anteaya | k thanks | 23:55 |
| mordred | jeblair: yay | 23:55 |
| jeblair | oh well, no new contributors | 23:55 |
| tjones | anteaya: yes | 23:55 |
| mordred | jeblair: didn't we land a patch to disable trivial rebase? | 23:55 |
| *** alexpilotti has joined #openstack-infra | 23:55 | |
| jeblair | mordred: yes, no one has investigated why it isn't working | 23:55 |
| *** aysyd has quit IRC | 23:55 | |
| fungi | jeblair: i tested updating my contact info earlier, after the upgrade, and it succeeded | 23:55 |
| jeblair | mordred: but we have other hooks | 23:55 |
| jeblair | fungi, mordred: that may be benign | 23:56 |
| jeblair | it happened before the upgrade too | 23:56 |
| fungi | jeblair: so either the 404 may be intermittent, or an actual e-mail address mismatch or similar | 23:56 |
| mordred | jeblair: ok | 23:56 |
| jeblair | fungi: with your confirmation it worked after the upgrade, i'll just ignore that for now | 23:56 |
| *** hogepodge has joined #openstack-infra | 23:57 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Actually disable trivial_rebase_hook https://review.openstack.org/90853 | 23:58 |
| mordred | jeblair, fungi: that should make trivial_rebase _ACTUALLY_ go away | 23:58 |
| anteaya | I'm fading, mattoliverau do you have enough of a sense of things you can take over answering random dev questions? | 23:59 |
| mattoliverau | anteaya: I'll do my best :) | 23:59 |
| jeblair | 2014-04-28 23:49:54,289 DEBUG zuul.Gerrit: Waiting for <Change 0x7fc8e017db90 86542,2> to appear in git repo | 23:59 |
| jeblair | 2014-04-28 23:54:57,512 DEBUG zuul.Gerrit: Change <Change 0x7fc8e017db90 86542,2> did not appear in the git repo | 23:59 |
| anteaya | mattoliverau: tag, you're it | 23:59 |
| jeblair | mordred: that's since the restart ^ | 23:59 |
| mordred | jeblair: blast | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
