| *** Apoorva_ has joined #openstack-infra | 00:00 | |
| *** cardeois has joined #openstack-infra | 00:03 | |
| *** Apoorva has quit IRC | 00:04 | |
| *** amitgandhinz has quit IRC | 00:05 | |
| *** cardeois has quit IRC | 00:10 | |
| kfox1111 | question... http://logs.openstack.org/66/386966/26/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/bb1e75a/console.html | 00:11 |
|---|---|---|
| kfox1111 | the job exited 0, but zuul then says the job completed with result FAILURE. any idea why that may happen? | 00:11 |
| fungi | (phrased in the form of a url) | 00:11 |
| *** cardeois has joined #openstack-infra | 00:11 | |
| kfox1111 | :) | 00:11 |
| fungi | this sounds similar to the issue mwhahaha linked. we're about to restart zuul with some extra debugging in place | 00:12 |
| kfox1111 | ah. k. | 00:12 |
| *** spzala has joined #openstack-infra | 00:13 | |
| *** tosky has quit IRC | 00:15 | |
| *** cardeois has quit IRC | 00:16 | |
| *** dfflanders has quit IRC | 00:17 | |
| jeblair | kfox1111, fungi, mordred: that error is interesting because while it does include the error we're working on addressing, the module_stdout includes a lot of broadcast messages from systemd-journal: http://logs.openstack.org/66/386966/26/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/bb1e75a/_zuul_ansible/ansible_log.txt | 00:17 |
| *** spzala has quit IRC | 00:18 | |
| jeblair | that may be a second problem. i wonder if it will cause ansible to fail to parse the output of the module it runs even after we fix the first error. | 00:18 |
| kfox1111 | looks like a bunch of messages from the ceph the job sets up. | 00:19 |
| kfox1111 | that part of the job hasn't changed in a while. | 00:20 |
| kfox1111 | some recent change in the ansible code is looking at it and choking maybe? | 00:21 |
| jeblair | yeah, we changed the way we run ansible; it's not immediately apparent to me what's going on though | 00:21 |
| jeblair | oh, i think this might be okay | 00:23 |
| *** tphummel has joined #openstack-infra | 00:23 | |
| jeblair | i think we're only getting that output now because the correct exception handler isn't being run. if it is, or if there is no exception, there should be no stdout/stderr in the ansible output. | 00:24 |
| jeblair | so i think that's a red herring | 00:24 |
| kfox1111 | k. | 00:24 |
| jeblair | we're probably just looking at the same problem again as fungi said | 00:24 |
| *** armax has joined #openstack-infra | 00:25 | |
| *** spzala has joined #openstack-infra | 00:26 | |
| jeblair | restarting now | 00:30 |
| kfox1111 | should I resubmit then? | 00:31 |
| kfox1111 | or wait for a bit? | 00:31 |
| jeblair | kfox1111: not yet | 00:31 |
| *** spzala has quit IRC | 00:32 | |
| kfox1111 | k. I'm going to head home now. I'll check back later. | 00:32 |
| kfox1111 | Thanks for the help. | 00:32 |
| jeblair | kfox1111: np, sorry for the inconvenience | 00:32 |
| kfox1111 | no worries. thanks, as always for all the hard work you do. :) | 00:33 |
| *** mtanino has quit IRC | 00:38 | |
| *** snarwade has quit IRC | 00:40 | |
| *** kashyap has quit IRC | 00:40 | |
| jeblair | restart complete | 00:41 |
| *** Julien-zte has joined #openstack-infra | 00:41 | |
| *** Julien-zte has quit IRC | 00:41 | |
| *** Julien-zte has joined #openstack-infra | 00:42 | |
| *** edmondsw has quit IRC | 00:46 | |
| *** xarses has joined #openstack-infra | 00:47 | |
| *** david-lyle_ has joined #openstack-infra | 00:51 | |
| *** Julien-zte has quit IRC | 00:52 | |
| *** baoli has joined #openstack-infra | 00:53 | |
| *** Julien-zte has joined #openstack-infra | 00:53 | |
| *** david-lyle has quit IRC | 00:54 | |
| *** gouthamr has joined #openstack-infra | 00:55 | |
| *** ijw has quit IRC | 00:56 | |
| *** ijw has joined #openstack-infra | 00:56 | |
| *** sree has joined #openstack-infra | 00:56 | |
| *** spzala has joined #openstack-infra | 00:59 | |
| *** sree has quit IRC | 01:00 | |
| *** amitgandhinz has joined #openstack-infra | 01:01 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-bandersnatch: Fix bandersnatch crons to support full sync https://review.openstack.org/383836 | 01:02 |
| *** zz_dimtruck is now known as dimtruck | 01:02 | |
| asselin | clarkb, finally got back to this ^^ | 01:02 |
| *** pahuang has quit IRC | 01:04 | |
| *** sflanigan has quit IRC | 01:05 | |
| *** armax has quit IRC | 01:06 | |
| *** ijw has quit IRC | 01:06 | |
| *** ijw has joined #openstack-infra | 01:06 | |
| *** armax has joined #openstack-infra | 01:06 | |
| *** armax has quit IRC | 01:08 | |
| *** armax has joined #openstack-infra | 01:10 | |
| *** dimtruck is now known as zz_dimtruck | 01:12 | |
| *** Apoorva_ has quit IRC | 01:12 | |
| *** armax has quit IRC | 01:14 | |
| *** Julien-zte has quit IRC | 01:15 | |
| *** Julien-zte has joined #openstack-infra | 01:15 | |
| *** chandanc has joined #openstack-infra | 01:17 | |
| *** gyee has quit IRC | 01:18 | |
| *** zhurong has joined #openstack-infra | 01:18 | |
| *** ijw has quit IRC | 01:19 | |
| *** ijw has joined #openstack-infra | 01:19 | |
| *** sflanigan has joined #openstack-infra | 01:20 | |
| *** sflanigan has joined #openstack-infra | 01:20 | |
| *** pahuang has joined #openstack-infra | 01:21 | |
| *** chandanc has quit IRC | 01:23 | |
| *** ijw has quit IRC | 01:24 | |
| *** thorst_ has quit IRC | 01:24 | |
| *** hongbin has joined #openstack-infra | 01:25 | |
| *** thorst_ has joined #openstack-infra | 01:25 | |
| *** ijw has joined #openstack-infra | 01:26 | |
| *** spzala has quit IRC | 01:26 | |
| *** sdake has quit IRC | 01:27 | |
| *** yanyanhu has joined #openstack-infra | 01:27 | |
| *** Julien-zte has quit IRC | 01:27 | |
| sc` | for anyone that was following along about rubygems.org, a point in time snapshot weighs in at 265gb. no idea what a regular delta would be | 01:29 |
| *** sdake has joined #openstack-infra | 01:29 | |
| *** mriedem_away is now known as mriedem | 01:33 | |
| *** thorst_ has quit IRC | 01:33 | |
| *** zz_dimtruck is now known as dimtruck | 01:35 | |
| *** amitgandhinz has quit IRC | 01:35 | |
| *** gildub has joined #openstack-infra | 01:36 | |
| *** kaisers_ has joined #openstack-infra | 01:37 | |
| *** maeker has quit IRC | 01:38 | |
| *** Julien-zte has joined #openstack-infra | 01:38 | |
| *** Julien-zte has quit IRC | 01:38 | |
| *** Julien-zte has joined #openstack-infra | 01:39 | |
| *** sdake has quit IRC | 01:40 | |
| *** mtanino has joined #openstack-infra | 01:41 | |
| *** kaisers_ has quit IRC | 01:41 | |
| *** ijw has quit IRC | 01:47 | |
| *** ijw has joined #openstack-infra | 01:47 | |
| *** r-daneel has quit IRC | 01:48 | |
| fungi | that sounds on par with a pypi mirror | 01:48 |
| *** aspiers has quit IRC | 01:53 | |
| *** ijw has quit IRC | 01:54 | |
| *** armax has joined #openstack-infra | 02:00 | |
| *** ijw has joined #openstack-infra | 02:03 | |
| *** yamamot__ has joined #openstack-infra | 02:05 | |
| *** yamamoto has quit IRC | 02:05 | |
| *** tqtran has quit IRC | 02:06 | |
| *** aspiers has joined #openstack-infra | 02:06 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver https://review.openstack.org/388918 | 02:10 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver https://review.openstack.org/388921 | 02:10 |
| *** ijw has quit IRC | 02:11 | |
| *** sflanigan has quit IRC | 02:13 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver https://review.openstack.org/388921 | 02:14 |
| *** thorst_ has joined #openstack-infra | 02:14 | |
| *** thorst_ has quit IRC | 02:14 | |
| *** ijw has joined #openstack-infra | 02:14 | |
| *** thorst_ has joined #openstack-infra | 02:14 | |
| *** armax has quit IRC | 02:15 | |
| *** mriedem has quit IRC | 02:17 | |
| *** ijw has quit IRC | 02:18 | |
| *** thorst_ has quit IRC | 02:23 | |
| *** sflanigan has joined #openstack-infra | 02:25 | |
| *** pahuang has quit IRC | 02:26 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove old weekly ArchWG meeting duplicate https://review.openstack.org/388708 | 02:27 |
| *** spzala has joined #openstack-infra | 02:27 | |
| *** amitgandhinz has joined #openstack-infra | 02:32 | |
| *** thorst_ has joined #openstack-infra | 02:32 | |
| *** thorst_ has quit IRC | 02:33 | |
| *** pahuang has joined #openstack-infra | 02:38 | |
| *** thorst_ has joined #openstack-infra | 02:39 | |
| *** thorst_ has quit IRC | 02:39 | |
| *** spzala has quit IRC | 02:43 | |
| *** baoli has quit IRC | 02:45 | |
| *** yuanying has quit IRC | 02:47 | |
| *** chandanc has joined #openstack-infra | 02:49 | |
| tonyb | Does anyone know about "[Zuul] standard output/error still open after child exited.* | [Zuul] Task exit code: 0" failures? | 02:55 |
| jamielennox | is there a reason that shade is not in g-r? | 02:56 |
| tonyb | As far as I can tell the devsatck/grenade/tox has passed but due to the message above the job fails | 02:57 |
| tonyb | jamielennox: It's not used by anythign that cares about co-installability? | 02:57 |
| tonyb | the errors seems to have started about Oct 11: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%5BZuul%5D%20standard%20output%2Ferror%20still%20open%20after%20child%5C%22%20AND%20message%3A%5C%22%7C%20%5BZuul%5D%20Task%20exit%20code%3A%200%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 02:58 |
| jamielennox | tonyb: it's just interesting that shade can be top of the tree when there are ansible modules and a whole bunch of infra stuff that uses it | 02:58 |
| *** yamahata has quit IRC | 02:59 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Don't set tracing in environment files https://review.openstack.org/388972 | 03:00 |
| openstackgerrit | yatin proposed openstack-infra/project-config: Add diskimage-builder to project list https://review.openstack.org/388973 | 03:01 |
| tonyb | jamielennox: I don;t have an objecttion but my limited understanding suggests that it wasn't really intented to be used inside OpenStack. Isn't the point of it multi-cloud compat which I'm not certain we handle within an OpenStack | 03:02 |
| *** knangia has quit IRC | 03:02 | |
| jamielennox | tonyb: ok, it was just interesting from an install from devstack perspective - it would seem like a conscious choice to not have it in | 03:03 |
| jamielennox | but yea, i guess nothing that is using it is being tested on openstack infra | 03:04 |
| *** amitgandhinz has quit IRC | 03:06 | |
| tonyb | jamielennox: yeah. | 03:07 |
| *** Julien-zte has quit IRC | 03:10 | |
| *** kashyap has joined #openstack-infra | 03:13 | |
| *** sdake has joined #openstack-infra | 03:14 | |
| *** rfolco has joined #openstack-infra | 03:17 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add -{node} to dsvm job names https://review.openstack.org/388893 | 03:19 |
| *** reed_ has joined #openstack-infra | 03:21 | |
| *** reed_ has quit IRC | 03:23 | |
| *** david-lyle_ has quit IRC | 03:23 | |
| *** david-lyle has joined #openstack-infra | 03:23 | |
| *** kaisers_ has joined #openstack-infra | 03:25 | |
| *** dave-mccowan has quit IRC | 03:27 | |
| *** mountpoint has quit IRC | 03:27 | |
| tonyb | given this failure: http://logs.openstack.org/8f/8f9f8de5a6177eb07ba067e78fade6c9ccde5df1/release-post/tag-releases/1823078/console.html does that imply that the rm of /etc/sudoers.d/jenkins-sudo failed and the job aborted? | 03:27 |
| *** vikrant has joined #openstack-infra | 03:27 | |
| * tonyb may have used his $random_questions budget | 03:27 | |
| *** coolsvap has joined #openstack-infra | 03:29 | |
| *** kaisers_ has quit IRC | 03:30 | |
| *** ramishra has quit IRC | 03:31 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/shade: Allow setting env variables for functional options https://review.openstack.org/388983 | 03:32 |
| *** ramishra has joined #openstack-infra | 03:32 | |
| *** ramishra_ has joined #openstack-infra | 03:35 | |
| *** ramishra has quit IRC | 03:37 | |
| *** thorst_ has joined #openstack-infra | 03:40 | |
| mgagne | tonyb: looks like it's running on signing01.ci.openstack.org which might not be ephemeral and sudo removed already or non-existant | 03:43 |
| *** krtaylor has joined #openstack-infra | 03:47 | |
| *** thorst_ has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-infra | 03:49 | |
| *** yamahata has joined #openstack-infra | 03:52 | |
| *** vikrant is now known as vikrant|brb | 03:53 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Introduce experimental dsvm jobs with xenial https://review.openstack.org/388985 | 03:53 |
| *** sree has joined #openstack-infra | 03:56 | |
| *** Julien-zte has joined #openstack-infra | 03:57 | |
| *** Julien-zte has quit IRC | 04:00 | |
| *** Julien-z_ has joined #openstack-infra | 04:00 | |
| *** sree has quit IRC | 04:00 | |
| *** tuanluong has joined #openstack-infra | 04:01 | |
| *** amitgandhinz has joined #openstack-infra | 04:02 | |
| *** hongbin has quit IRC | 04:03 | |
| *** tuanluong has quit IRC | 04:03 | |
| *** rfolco has quit IRC | 04:03 | |
| *** tuanluong has joined #openstack-infra | 04:05 | |
| *** mtanino has quit IRC | 04:05 | |
| *** Julien-z_ has quit IRC | 04:06 | |
| *** Julien-zte has joined #openstack-infra | 04:06 | |
| *** sdake has quit IRC | 04:07 | |
| *** Julien-zte has quit IRC | 04:08 | |
| *** Julien-z_ has joined #openstack-infra | 04:08 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/shade: Add a devstack plugin for shade https://review.openstack.org/388988 | 04:10 |
| *** sdake has joined #openstack-infra | 04:13 | |
| *** maeker has joined #openstack-infra | 04:16 | |
| *** csomerville has quit IRC | 04:19 | |
| *** csomerville has joined #openstack-infra | 04:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn down yum install-packages https://review.openstack.org/388990 | 04:20 |
| *** netsin has quit IRC | 04:22 | |
| *** links has joined #openstack-infra | 04:23 | |
| *** cody-somerville has joined #openstack-infra | 04:23 | |
| *** cody-somerville has joined #openstack-infra | 04:23 | |
| *** csomerville has quit IRC | 04:25 | |
| *** chandanc has quit IRC | 04:25 | |
| *** sdake has quit IRC | 04:26 | |
| *** links has quit IRC | 04:27 | |
| *** pgadiya has joined #openstack-infra | 04:28 | |
| *** ijw has joined #openstack-infra | 04:31 | |
| ramishra_ | ianw: hey around? | 04:34 |
| *** ijw has quit IRC | 04:35 | |
| ramishra_ | ianw: https://bugs.launchpad.net/heat/+bug/1635111, it seems all grenade jobs are failing. | 04:36 |
| openstack | Launchpad bug 1635111 in heat "grenade jobs are failing with 'standard output/error still open after child exited'" [Undecided,New] | 04:36 |
| *** amitgandhinz has quit IRC | 04:37 | |
| ramishra_ | not sure, but looks like this is due some recent zuul changes. | 04:37 |
| ramishra_ | fungi: hi ^^^ | 04:38 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/devstack-gate: DNM: For Debugging only https://review.openstack.org/388995 | 04:40 |
| *** spzala has joined #openstack-infra | 04:44 | |
| *** thorst_ has joined #openstack-infra | 04:48 | |
| *** ijw has joined #openstack-infra | 04:48 | |
| *** spzala has quit IRC | 04:48 | |
| *** markvoelker_ has quit IRC | 04:49 | |
| *** ijw has quit IRC | 04:53 | |
| *** thorst_ has quit IRC | 04:54 | |
| *** rajinir has quit IRC | 04:56 | |
| *** baoli has joined #openstack-infra | 04:57 | |
| *** baoli has quit IRC | 05:01 | |
| *** yamamot__ has quit IRC | 05:04 | |
| *** jaosorior has joined #openstack-infra | 05:09 | |
| *** caowei has joined #openstack-infra | 05:14 | |
| *** kaisers_ has joined #openstack-infra | 05:15 | |
| *** kaisers_ has quit IRC | 05:19 | |
| *** maeker has quit IRC | 05:20 | |
| *** bhavik has joined #openstack-infra | 05:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: don't close stdout in command module https://review.openstack.org/389009 | 05:26 |
| *** bhavik has quit IRC | 05:27 | |
| jeblair | ramishra_, kfox1111, mwhahaha, fungi, mordred, ianw, infra-root: ^ my earlier change to fix the get_exception error ( https://review.openstack.org/388936 ) revealed this error: "close() called during concurrent operation on the same file object." ( http://logs.openstack.org/89/388589/1/check/gate-puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/33ddee2/_zuul_ansible/ansible_log.txt ). i *think* that ... | 05:29 |
| jeblair | ... https://review.openstack.org/389009 will fix the problem. however, i am not able to restart the launchers with that change now. | 05:29 |
| *** markvoelker_ has joined #openstack-infra | 05:30 | |
| jeblair | infra-root: however, if the next person able would like to land that and restart, (or else land a revert change of the command module work) that would be great. | 05:30 |
| ramishra_ | jeblair: I can do the revert if that helps. | 05:32 |
| *** amitgandhinz has joined #openstack-infra | 05:33 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/zuul: Revert "Ansible launcher: import get_exception in ansible command" https://review.openstack.org/389013 | 05:34 |
| jeblair | ramishra_: mordred prepared https://review.openstack.org/388004 though it is now out of date. if you want to update it that might be helpful (but the root that decides to restart may decide to push forward rather than backward) | 05:34 |
| jeblair | ramishra_: oh, that commit isn't the problem -- that commit just allowed us to actually *see* the problem | 05:35 |
| jeblair | ramishra_: the commit that created the problem is Iae4769f923ecf74462e1fe43168ea93ff1c61d6e (but probably all of those commits should be reverted because they were all tested together) | 05:36 |
| ramishra_ | jeblair: yeah I undersatsand that, I thought we would revert it and then fix the issue and merge itagain. | 05:36 |
| ramishra_ | jeblair: would not that help? | 05:37 |
| jeblair | ramishra_: no, if we wanted to revert, we would do something like https://review.openstack.org/388004 (but as i said, it needs updating) | 05:38 |
| sc` | with the magic number for rubygems.org known, what would need to happen in order to get some momentum on https://review.openstack.org/253616 ? | 05:38 |
| ramishra_ | jeblair: ok, sorry, I've little understanding of these things. Ok, I'll try and push a change to revert all command module related changes. | 05:39 |
| *** jtomasek has quit IRC | 05:39 | |
| jeblair | anyway, i'm sorry i have to go (it's quite late here and it takes about 30-60 minutes to restart the launchers). hopefully an infra-root in sunlight can fix this soon. | 05:40 |
| sc` | crap. it _is_ late. enough about computers for the night o/ | 05:42 |
| *** asselin has quit IRC | 05:45 | |
| *** yamamoto has joined #openstack-infra | 05:45 | |
| *** asselin__ has joined #openstack-infra | 05:46 | |
| *** markvoelker_ has quit IRC | 05:46 | |
| *** camunoz has quit IRC | 05:48 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 05:50 |
| *** thorst_ has joined #openstack-infra | 05:52 | |
| *** hichihara has joined #openstack-infra | 05:53 | |
| *** thorst_ has quit IRC | 05:59 | |
| *** hurgleburgler has quit IRC | 06:01 | |
| *** aeng has quit IRC | 06:01 | |
| *** amitgandhinz has quit IRC | 06:07 | |
| *** dimtruck is now known as zz_dimtruck | 06:09 | |
| *** Naeil has joined #openstack-infra | 06:11 | |
| mordred | tonybm jamielennox: I don't think there are any issues with shade being in g-r - but also what you said is accurate, it's not intended to be used _by_ openstack as much as it's intended to be used _on_ openstack | 06:12 |
| *** zz_dimtruck is now known as dimtruck | 06:12 | |
| mordred | jeblair: dude. you were up way too late | 06:14 |
| mordred | jeblair: I think I'm caught up on scrollback now | 06:14 |
| *** e0ne has joined #openstack-infra | 06:15 | |
| mordred | infra-root: it's 1am, so I'm not going to start a launcher restart since it's unlikely it'll finish before I fall asleep. I did approve jeblair's patch because I think rolling it out before reverting is the right step to take. I'll run a restart first thing in the morning if nobody has beaten me to it | 06:16 |
| mordred | but as soon as https://review.openstack.org/389009 and puppet rolls it out to launchers, running the restart playbook should be fine | 06:17 |
| mordred | there is a copy of it in ~root on puppetmaster - ls -ltra should show it to you | 06:17 |
| *** chandankumar has joined #openstack-infra | 06:18 | |
| *** pcaruana has joined #openstack-infra | 06:18 | |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: don't close stdout in command module https://review.openstack.org/389009 | 06:19 |
| *** tqtran has joined #openstack-infra | 06:19 | |
| *** yolanda has quit IRC | 06:20 | |
| *** andreas_s has joined #openstack-infra | 06:20 | |
| *** tqtran has quit IRC | 06:23 | |
| *** e0ne has quit IRC | 06:29 | |
| *** e0ne has joined #openstack-infra | 06:29 | |
| jaosorior | mordred: any estimate on how long will it take for this to come into effect https://review.openstack.org/389009 ? | 06:30 |
| *** tphummel has quit IRC | 06:32 | |
| *** florianf has joined #openstack-infra | 06:33 | |
| *** kaisers_ has joined #openstack-infra | 06:35 | |
| *** nherciu has joined #openstack-infra | 06:36 | |
| *** sree has joined #openstack-infra | 06:37 | |
| *** migi_ is now known as migi | 06:38 | |
| *** vsaienko has joined #openstack-infra | 06:38 | |
| *** sputnik13 has joined #openstack-infra | 06:40 | |
| *** yanyanhu has quit IRC | 06:42 | |
| *** vsaienko has quit IRC | 06:42 | |
| *** vsaienko has joined #openstack-infra | 06:43 | |
| *** zhurong_ has joined #openstack-infra | 06:45 | |
| *** zhurong has quit IRC | 06:45 | |
| *** tphummel has joined #openstack-infra | 06:46 | |
| mordred | jaosorior: we're waiting on someone to be awake enough to run a zuul-launcher rolling restart | 06:47 |
| *** yanyanhu has joined #openstack-infra | 06:54 | |
| *** aviau has quit IRC | 06:55 | |
| *** tphummel has quit IRC | 06:55 | |
| *** aviau has joined #openstack-infra | 06:55 | |
| *** dimtruck is now known as zz_dimtruck | 06:56 | |
| *** vsaienko has quit IRC | 06:57 | |
| *** thorst_ has joined #openstack-infra | 06:58 | |
| *** yanyanhu_ has joined #openstack-infra | 07:00 | |
| *** martinkopec has joined #openstack-infra | 07:02 | |
| *** gildub has quit IRC | 07:02 | |
| *** yanyanhu has quit IRC | 07:04 | |
| *** amitgandhinz has joined #openstack-infra | 07:04 | |
| *** thorst_ has quit IRC | 07:04 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 07:05 |
| *** automagically has quit IRC | 07:06 | |
| *** automagically has joined #openstack-infra | 07:06 | |
| *** flepied has quit IRC | 07:08 | |
| *** pahuang has quit IRC | 07:09 | |
| *** tqtran has joined #openstack-infra | 07:10 | |
| *** dtardivel has joined #openstack-infra | 07:11 | |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** tesseract is now known as Guest14069 | 07:12 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: use ubuntu-xenial for newton+ https://review.openstack.org/385036 | 07:15 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: add periodic tempest job for stable branches https://review.openstack.org/385208 | 07:15 |
| *** claudiub has joined #openstack-infra | 07:16 | |
| *** vsaienko has joined #openstack-infra | 07:19 | |
| *** amoralej|off is now known as amoralej | 07:21 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: add periodic tempest job for stable branches https://review.openstack.org/385208 | 07:21 |
| *** tkelsey has joined #openstack-infra | 07:21 | |
| *** esikachev has joined #openstack-infra | 07:22 | |
| *** tnovacik has quit IRC | 07:25 | |
| *** oanson has joined #openstack-infra | 07:26 | |
| *** amitgandhinz has quit IRC | 07:27 | |
| *** ijw has joined #openstack-infra | 07:27 | |
| *** spzala has joined #openstack-infra | 07:29 | |
| *** spzala has quit IRC | 07:34 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** yaume has joined #openstack-infra | 07:37 | |
| *** vsaienko has quit IRC | 07:39 | |
| *** ijw has quit IRC | 07:41 | |
| *** Julien-z_ has quit IRC | 07:50 | |
| *** Julien-zte has joined #openstack-infra | 07:51 | |
| ianw | ummm, let me see ... | 07:52 |
| *** vsaienko has joined #openstack-infra | 07:53 | |
| *** flepied has joined #openstack-infra | 07:53 | |
| *** flepied has quit IRC | 07:53 | |
| *** flepied1 has joined #openstack-infra | 07:54 | |
| *** sdake has joined #openstack-infra | 07:54 | |
| *** jordanP has joined #openstack-infra | 07:54 | |
| *** sdake has quit IRC | 07:55 | |
| *** sdake has joined #openstack-infra | 07:56 | |
| ianw | jeblair / mordred / anyone: well, i'm running the playbook in a root screen session. it seems to be working | 07:57 |
| *** yamahata has quit IRC | 07:57 | |
| *** yanyanhu_ has quit IRC | 07:57 | |
| *** pilgrimstack has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** ijw has joined #openstack-infra | 08:00 | |
| *** vsaienko has quit IRC | 08:03 | |
| *** thorst_ has joined #openstack-infra | 08:03 | |
| *** Julien-zte has quit IRC | 08:04 | |
| *** jpich has joined #openstack-infra | 08:04 | |
| ianw | p.s. i did check 388936 had rolled out, seems to be there on the launchers | 08:05 |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 08:05 |
| *** Julien-zte has joined #openstack-infra | 08:05 | |
| *** ijw has quit IRC | 08:05 | |
| *** sree_ has joined #openstack-infra | 08:05 | |
| *** yatinkarel has joined #openstack-infra | 08:05 | |
| *** sree_ is now known as Guest71336 | 08:06 | |
| *** david-lyle_ has joined #openstack-infra | 08:07 | |
| *** e0ne has quit IRC | 08:08 | |
| *** sree has quit IRC | 08:08 | |
| *** oanson has quit IRC | 08:09 | |
| *** david-lyle has quit IRC | 08:09 | |
| *** thorst_ has quit IRC | 08:09 | |
| *** mhickey has joined #openstack-infra | 08:11 | |
| *** qwertyco has joined #openstack-infra | 08:11 | |
| *** yanyanhu_ has joined #openstack-infra | 08:14 | |
| *** dizquierdo has joined #openstack-infra | 08:15 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add Barbican key order to scenario002 https://review.openstack.org/389057 | 08:15 |
| *** markvoelker has joined #openstack-infra | 08:19 | |
| *** sflanigan has quit IRC | 08:19 | |
| *** ccamacho|afk is now known as ccamacho | 08:19 | |
| *** tqtran has quit IRC | 08:21 | |
| *** amitgandhinz has joined #openstack-infra | 08:23 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 08:26 |
| *** dingyichen has quit IRC | 08:28 | |
| openstackgerrit | Waldemar Znoinski proposed openstack-infra/project-config: gerrit: add intel-nfv-ci-tests-ci group https://review.openstack.org/388894 | 08:31 |
| *** ijw has joined #openstack-infra | 08:33 | |
| *** flepied1 is now known as flepied | 08:34 | |
| *** Julien-zte has quit IRC | 08:35 | |
| *** r1chardj0n3s has joined #openstack-infra | 08:36 | |
| *** Julien-zte has joined #openstack-infra | 08:36 | |
| r1chardj0n3s | hi folks. We're seeing pretty consistent failures in the gate for Horizon jobs, related to xvfb, I think. An example: http://logs.openstack.org/44/388244/2/check/gate-horizon-nodejs4-npm-run-lint/8057ad4/console.html | 08:37 |
| *** vsaienko has joined #openstack-infra | 08:38 | |
| *** ijw has quit IRC | 08:38 | |
| *** tosky has joined #openstack-infra | 08:39 | |
| openstackgerrit | Merged openstack/os-client-config: Update ECS image_api_version to 1 https://review.openstack.org/388901 | 08:40 |
| docaedo | r1chardj0n3s: Looks to me like the gate is broken (zuul change), bug here: https://bugs.launchpad.net/zuul/+bug/1635111 | 08:41 |
| openstack | Launchpad bug 1635111 in Zuul "All grenade jobs are failing with error" [High,New] | 08:41 |
| r1chardj0n3s | thanks docaedo | 08:41 |
| *** hichihara has quit IRC | 08:41 | |
| vsaienk0 | infra-team, could you please help with http://logs.openstack.org/11/374811/2/check/gate-grenade-dsvm-ironic/9328d49/console.html#_2016-10-20_08_08_21_824684 all tests are passed but the job is failed. It looks like it was caused by timeout, but it is strange job timeout is 180 min, and it took near 60 min so I'm confused | 08:41 |
| frickler | r1chardj0n3s: docaedo: seems https://review.openstack.org/389009 is the fix and iiuc ianw has just rolled it out, so maybe try a recheck | 08:42 |
| r1chardj0n3s | thanks frickler, will give it a go | 08:42 |
| *** spzala has joined #openstack-infra | 08:43 | |
| *** tosky has joined #openstack-infra | 08:45 | |
| *** electrofelix has joined #openstack-infra | 08:45 | |
| *** derekh has joined #openstack-infra | 08:46 | |
| *** spzala has quit IRC | 08:47 | |
| *** pblaho has joined #openstack-infra | 08:48 | |
| *** vsaienko has quit IRC | 08:48 | |
| *** Julien-zte has quit IRC | 08:48 | |
| *** Julien-zte has joined #openstack-infra | 08:49 | |
| *** otherwiseguy has quit IRC | 08:50 | |
| ianw | frickler / r1chardj0n3s : it's rolling out ... i'm not sure how long to wait for each zuul-launcher to stop being my first time doing this ... i'll give it a bit before i manually intervene | 08:50 |
| *** Julien-zte has joined #openstack-infra | 08:50 | |
| r1chardj0n3s | ok thanks ianw | 08:50 |
| *** jbernard has quit IRC | 08:51 | |
| *** jbernard has joined #openstack-infra | 08:51 | |
| docaedo | ianw: thanks - at least one patch that was failing earlier for me (with the sudo issue) is good now | 08:51 |
| *** otherwiseguy has joined #openstack-infra | 08:52 | |
| *** esikachev has quit IRC | 08:54 | |
| *** woodster_ has quit IRC | 08:55 | |
| *** dtantsur|sick is now known as dtantsur | 08:56 | |
| ianw | jeblair / mordred : ok, i had to intervene in zl06 & zl02 which seemed to get stuck. i'll leave the screen session on puppetmaster. monitoring for now, but otherwise zl0[1-7] report they restarted ok | 08:57 |
| *** amitgandhinz has quit IRC | 08:57 | |
| ajafo | hi, guys, who can help us with http://lists.openstack.org/pipermail/openstack-infra/2016-October/004784.html we need to add cross-repos core group or first core to the group? | 08:59 |
| rcarrillocruz | ajafo: fuel-ccp-ceph-core has now fule-ccp-core included | 09:01 |
| ajafo | rcarrillocruz: thanks | 09:01 |
| *** Julien-zte has quit IRC | 09:02 | |
| *** Julien-zte has joined #openstack-infra | 09:03 | |
| *** e0ne has joined #openstack-infra | 09:03 | |
| *** Rockyg has quit IRC | 09:06 | |
| *** thorst_ has joined #openstack-infra | 09:07 | |
| *** sambetts|afk is now known as sambetts | 09:10 | |
| *** sdake has quit IRC | 09:11 | |
| *** jordanP has quit IRC | 09:12 | |
| *** jordanP has joined #openstack-infra | 09:13 | |
| *** tnovacik has joined #openstack-infra | 09:13 | |
| *** Julien-zte has quit IRC | 09:13 | |
| therve | Grenade heat gate is failing bizarrely: http://logs.openstack.org/23/388723/2/check/gate-grenade-dsvm-heat/a360994/console.html | 09:13 |
| therve | The only error I can see is "standard output/error still open after child exited", does that remind somebody something? | 09:13 |
| *** Julien-zte has joined #openstack-infra | 09:14 | |
| *** thorst_ has quit IRC | 09:14 | |
| docaedo | therve: believe https://review.openstack.org/389009 was the fix, which ianw has been rolling out to the zuul launches | 09:15 |
| docaedo | therve: also https://bugs.launchpad.net/zuul/+bug/1635111 | 09:15 |
| openstack | Launchpad bug 1635111 in Zuul "All grenade jobs are failing with error" [High,New] | 09:15 |
| therve | Ah yeah that's the one | 09:16 |
| therve | docaedo, Is the fix taking some time to roll out? | 09:16 |
| *** panda|Zz is now known as panda | 09:18 | |
| *** ihrachys has joined #openstack-infra | 09:19 | |
| docaedo | therve: I think that last response on 388723 from an hour ago is right about when the fix was rolling out - I had 5 patches that passed re-check in the last half hour | 09:20 |
| therve | docaedo, Ok, thanks a lot | 09:20 |
| docaedo | therve: no prob | 09:20 |
| *** vsaienko has joined #openstack-infra | 09:21 | |
| *** jtomasek_ has joined #openstack-infra | 09:21 | |
| *** john-davidge has joined #openstack-infra | 09:22 | |
| *** mfedosin has joined #openstack-infra | 09:22 | |
| *** oanson has joined #openstack-infra | 09:23 | |
| *** john-davidge has quit IRC | 09:24 | |
| *** john-davidge has joined #openstack-infra | 09:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add Barbican key order to scenario002 https://review.openstack.org/389057 | 09:30 |
| gmann | AJaeger: for you - https://review.openstack.org/#/c/372581/ | 09:31 |
| *** matrohon has joined #openstack-infra | 09:34 | |
| *** jtomasek_ is now known as jtomasek | 09:35 | |
| *** zhurong_ has quit IRC | 09:35 | |
| *** Hal has quit IRC | 09:38 | |
| *** Hal has joined #openstack-infra | 09:38 | |
| *** markvoelker has quit IRC | 09:39 | |
| *** spzala has joined #openstack-infra | 09:41 | |
| *** derekh has quit IRC | 09:43 | |
| panda | does anyonw has any ides why build-timeout wrapper for tripleo job-template is not working ? all our jobs are started with a timeout of 110 instead of 180 | 09:43 |
| panda | anyone* idea* | 09:43 |
| *** jaosorior has quit IRC | 09:44 | |
| *** r1chardj0n3s has left #openstack-infra | 09:44 | |
| *** jaosorior has joined #openstack-infra | 09:44 | |
| pabelanger | panda: it is possible there is a regression in our most recent zuul update | 09:45 |
| pabelanger | panda: do you have a log file? | 09:45 |
| *** spzala has quit IRC | 09:45 | |
| panda | pabelanger: is this enough ? http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-mitaka/960f689/console.html#_2016-10-20_06_03_11_007685 | 09:46 |
| *** derekh has joined #openstack-infra | 09:47 | |
| panda | pabelanger: were you looking for a different type of log ? | 09:47 |
| pabelanger | panda: that works, thanks | 09:47 |
| panda | pabelanger: all tripleo jobs are affected | 09:47 |
| pabelanger | I think we are no longer setting the timeout environment variable properly | 09:47 |
| pabelanger | will know more in a few mins | 09:48 |
| *** caowei has quit IRC | 09:49 | |
| panda | pabelanger: thanks | 09:50 |
| *** tuanluong has quit IRC | 09:50 | |
| pabelanger | Ya, looks like we are not longer exposing it | 09:53 |
| pabelanger | working on a patch | 09:53 |
| *** amitgandhinz has joined #openstack-infra | 09:53 | |
| panda | pabelanger: great :) | 09:54 |
| *** Julien-zte has quit IRC | 09:55 | |
| amoralej | i'm still getting jobs failing with "standard output/error still open after child exited ..." is it expected? | 09:55 |
| *** caowei has joined #openstack-infra | 09:56 | |
| panda | amoralej: is it the error just at the end of the job? | 09:56 |
| *** ssbarnea has joined #openstack-infra | 09:56 | |
| amoralej | yes | 09:56 |
| amoralej | http://logs.openstack.org/47/389047/1/check/gate-puppet-openstack-integration-3-scenario002-tempest-centos-7/61644ac/console.html | 09:56 |
| amoralej | jobs seems to have run ok, but i'm getting failure | 09:57 |
| amoralej | after two rechecks i reduced from 4 failures to 2 failures, but still getting them | 09:57 |
| amoralej | this is from 10 minutes ago | 09:57 |
| panda | amoralej: it happens also when job succeeds, look pretty harmless error. | 09:58 |
| *** caowei has quit IRC | 09:58 | |
| *** amotoki has joined #openstack-infra | 09:58 | |
| amoralej | yeah, i'm not worried about the error message but about the job false failure | 09:59 |
| ianw | amoralej: it's not http://logs.openstack.org/47/389047/1/check/gate-puppet-openstack-integration-3-scenario002-tempest-centos-7/61644ac/console.html#_2016-10-20_09_36_02_573757 ? | 09:59 |
| amoralej | that 2 is expected | 09:59 |
| amoralej | from a successfull one http://logs.openstack.org/47/389047/1/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/a85d9f3/console.html#_2016-10-20_09_35_52_901191 | 10:00 |
| *** Guest71336 has quit IRC | 10:03 | |
| ianw | amoralej: hmm, interesting ... it seems that the zuul change 389009 is *not* rolled out on zl01 | 10:04 |
| slagle | i'm seeing an error during the ansible run at the end of otherwise successful jobs that is causing them to fail: http://logs.openstack.org/57/389057/2/check/gate-tripleo-ci-centos-7-undercloud/cecc845/_zuul_ansible/ansible_log.txt | 10:04 |
| openstackgerrit | Jordan Pittier proposed openstack-infra/shade: Logging: avoid string interpolation when not needed https://review.openstack.org/389104 | 10:04 |
| slagle | amoralej: looks like the same thing you might be seeing | 10:04 |
| ianw | slagle: yeah, that's also zl01 launcher | 10:05 |
| ianw | i wonder if it's not puppeting | 10:05 |
| *** ociuhandu has quit IRC | 10:06 | |
| amoralej | ok ianw , let me know when change is rolled out on it and i'll recheck | 10:06 |
| amoralej | thanks | 10:06 |
| *** lezbar has quit IRC | 10:08 | |
| *** winggundamth has quit IRC | 10:08 | |
| *** ldnunes has joined #openstack-infra | 10:09 | |
| *** winggundamth has joined #openstack-infra | 10:10 | |
| *** yanyanhu_ has quit IRC | 10:10 | |
| *** lezbar has joined #openstack-infra | 10:11 | |
| *** thorst_ has joined #openstack-infra | 10:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Add back timeout_var logic https://review.openstack.org/389108 | 10:14 |
| pabelanger | panda: mordred: ^ that will fix devstack-gate timeout issues | 10:14 |
| *** vsaienko has quit IRC | 10:19 | |
| *** thorst_ has quit IRC | 10:19 | |
| panda | pabelanger: thanks, that was fast :) | 10:19 |
| *** degorenko|afk is now known as degorenko | 10:20 | |
| *** dizquierdo is now known as dizquierdo_afk | 10:20 | |
| *** sflanigan has joined #openstack-infra | 10:22 | |
| panda | pabelanger: does that change means that there is a different way to get a devstack timeout now ? | 10:22 |
| *** ralonsoh has joined #openstack-infra | 10:22 | |
| panda | pabelanger: I mean this one Ie51de4a135d953c4ad9dcb773d27b3c54ca8829b that removed the timeout_var | 10:22 |
| *** jordanP has quit IRC | 10:27 | |
| *** amitgandhinz has quit IRC | 10:27 | |
| *** rossella_s has quit IRC | 10:28 | |
| *** rossella_s has joined #openstack-infra | 10:28 | |
| *** TomazVieira has quit IRC | 10:29 | |
| openstackgerrit | Merged openstack-infra/release-tools: ensure version numbers are always strings https://review.openstack.org/388813 | 10:31 |
| *** derekh has quit IRC | 10:31 | |
| openstackgerrit | Merged openstack-infra/release-tools: ensure all proposed versions include major.minor.patch values https://review.openstack.org/388814 | 10:31 |
| *** tlian has quit IRC | 10:31 | |
| *** gildub has joined #openstack-infra | 10:33 | |
| ianw | TASK [puppet : set log filename] *********************************************** | 10:34 |
| ianw | ERROR! Unexpected Exception: [Errno 12] Cannot allocate memory | 10:34 |
| ianw | to see the full traceback, use -vvv | 10:34 |
| ianw | pabelanger: ^ that doesn't look promising | 10:34 |
| *** derekh has joined #openstack-infra | 10:34 | |
| ianw | pabelanger: yeah, i think puppet run's aren't getting all the way through, that's in puppetmaster puppet_run_all_cron.log | 10:35 |
| ianw | [31016926.950503] Killed process 1794 (ansible-playboo) total-vm:1163856kB, anon-rss:122312kB, file-rss:252kB | 10:36 |
| ianw | umm, am i nuts or is it a 2gb host? | 10:37 |
| *** markvoelker has joined #openstack-infra | 10:37 | |
| *** ijw has joined #openstack-infra | 10:38 | |
| rcarrillocruz | heh | 10:39 |
| rcarrillocruz | yeah | 10:39 |
| rcarrillocruz | it's a long standing item to replace it iirc | 10:39 |
| ianw | rcarrillocruz: i think i know what you're doing today :) | 10:40 |
| * rcarrillocruz walks away misteriously | 10:40 | |
| ianw | i'm not sure there's much to do other than upsize it | 10:40 |
| ianw | i'm manually running puppet for zl* hosts now ... hopefully just running on the reduced host list will be ok | 10:41 |
| ianw | that should hopefully redeploy zuul on zl01, which i'll restart, and fix up the last of these issues | 10:41 |
| *** ijw has quit IRC | 10:43 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add Barbican key order to scenario002 https://review.openstack.org/389057 | 10:44 |
| *** amotoki has quit IRC | 10:46 | |
| ianw | ahhh, zl01 is in the emergency file! that's not helping | 10:47 |
| rcarrillocruz | i'm looking at cacti, the create_graphs.sh is broken, it pulls fields that are present on vanilla which are not for some reason on chocolate | 10:50 |
| rcarrillocruz | which makes the whole tree of chocolate to not appear | 10:50 |
| rcarrillocruz | sigh | 10:50 |
| *** jkilpatr has quit IRC | 10:51 | |
| ianw | pabelanger / jeblair / mordred : i think i'm going to tap out. i'm assuming one of you put zl01 in the emergency file, so 389009 is not applied there. although it would probably be fine, i'm not up for debugging it ATM should it explode if i manually run. otherwise change is rolled out | 10:51 |
| *** gmann__ has joined #openstack-infra | 10:51 | |
| ianw | pabelanger / jeblair / mordred : and yeah, the oom's cause by ansible-playbook on puppetmaster probably need corrective action ... looks like the cron job hits it frequently | 10:52 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Fix iface variable name on create_graphs.sh loop https://review.openstack.org/389118 | 10:52 |
| rcarrillocruz | ianw: i believe mordred did (the placement of zl01 on emergency) | 10:52 |
| *** ociuhandu has joined #openstack-infra | 10:55 | |
| *** tnovacik has quit IRC | 10:57 | |
| *** liusheng has quit IRC | 10:58 | |
| *** dprince has joined #openstack-infra | 10:58 | |
| rcarrillocruz | btw, i've added cacti01 to emergency, till we land ^ | 10:59 |
| rcarrillocruz | as i put the hotfix on the script | 10:59 |
| rcarrillocruz | otherwise puppet will overwrite it | 10:59 |
| *** Julien-zte has joined #openstack-infra | 11:02 | |
| *** dizquierdo_afk is now known as dizquierdo | 11:02 | |
| *** Rockyg has joined #openstack-infra | 11:03 | |
| rcarrillocruz | thx ianw , i'll approve now | 11:05 |
| *** kairat has joined #openstack-infra | 11:06 | |
| ianw | #status jobs launched by zl01.openstack.org (check console.html) may fail due to 389009 | 11:06 |
| *** markvoelker_ has joined #openstack-infra | 11:07 | |
| kairat | fungi, hello, could you please look at https://review.openstack.org/#/c/359029/. It would allow us to test deployment of new app-catalog with Glare on staging. | 11:07 |
| ianw | alright, heading out for tonight, good luck all :) | 11:08 |
| rcarrillocruz | have a good one ianw | 11:08 |
| *** lucas-sick is now known as lucasagomes | 11:10 | |
| *** markvoelker has quit IRC | 11:11 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix iface variable name on create_graphs.sh loop https://review.openstack.org/389118 | 11:13 |
| *** jkilpatr has joined #openstack-infra | 11:14 | |
| *** thorst_ has joined #openstack-infra | 11:17 | |
| *** thorst_ has quit IRC | 11:17 | |
| *** sputnik13 has quit IRC | 11:17 | |
| *** thorst_ has joined #openstack-infra | 11:17 | |
| *** zeih has joined #openstack-infra | 11:19 | |
| *** admcleod- has joined #openstack-infra | 11:19 | |
| pabelanger | panda: Ya, we'll want to change the way we configure it for zuulv3, but that won't happen for a while yet | 11:20 |
| pabelanger | panda: yes, we need to revert parts of Ie51de4a135d953c4ad9dcb773d27b3c54ca8829b | 11:21 |
| panda | pabelanger: ack thanks. | 11:21 |
| pabelanger | ianw: that is, expected. We have OOM issues on puppetmaster.o.o and need to deploy a new server | 11:21 |
| *** mtanino has joined #openstack-infra | 11:23 | |
| *** amitgandhinz has joined #openstack-infra | 11:24 | |
| *** erlon has joined #openstack-infra | 11:29 | |
| *** claudiub|2 has joined #openstack-infra | 11:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge https://review.openstack.org/389127 | 11:30 |
| *** claudiub has quit IRC | 11:33 | |
| *** baoli has joined #openstack-infra | 11:36 | |
| *** ccamacho is now known as ccamacho|lunch | 11:39 | |
| *** baoli has quit IRC | 11:40 | |
| *** qwertyco has quit IRC | 11:42 | |
| *** tiswanso has joined #openstack-infra | 11:42 | |
| *** baoli has joined #openstack-infra | 11:42 | |
| *** gouthamr has quit IRC | 11:43 | |
| *** krtaylor has quit IRC | 11:43 | |
| *** jpena is now known as jpena|lunch | 11:43 | |
| *** tiswanso has quit IRC | 11:47 | |
| *** dizquierdo has quit IRC | 11:47 | |
| *** nicolasbock has joined #openstack-infra | 11:49 | |
| *** ssbarnea has quit IRC | 11:51 | |
| *** ssbarnea has joined #openstack-infra | 11:52 | |
| *** gouthamr has joined #openstack-infra | 11:55 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 11:58 |
| *** tiswanso has joined #openstack-infra | 11:58 | |
| *** amitgandhinz has quit IRC | 11:58 | |
| *** gouthamr has quit IRC | 11:59 | |
| *** baoli_ has joined #openstack-infra | 12:00 | |
| *** gouthamr has joined #openstack-infra | 12:00 | |
| *** gildub has quit IRC | 12:00 | |
| AJaeger | pabelanger: arrived well in Paris? | 12:01 |
| *** baoli has quit IRC | 12:03 | |
| *** tiswanso_ has joined #openstack-infra | 12:05 | |
| *** tiswanso has quit IRC | 12:07 | |
| *** edmondsw has joined #openstack-infra | 12:08 | |
| *** zhurong has joined #openstack-infra | 12:08 | |
| *** amoralej is now known as amoralej|lunch | 12:08 | |
| *** markvoelker has joined #openstack-infra | 12:12 | |
| *** tiswanso has joined #openstack-infra | 12:13 | |
| *** tiswanso_ has quit IRC | 12:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Add diskimage-builder to project list https://review.openstack.org/388973 | 12:15 |
| *** oanson has quit IRC | 12:15 | |
| *** markvoelker_ has quit IRC | 12:16 | |
| *** EricGonczer_ has joined #openstack-infra | 12:16 | |
| *** tiswanso_ has joined #openstack-infra | 12:19 | |
| *** tiswanso has quit IRC | 12:21 | |
| pabelanger | AJaeger: yes, in the Red Hat office now | 12:22 |
| pabelanger | rcarrillocruz: crinkle: I've added a few more slides to our presentation | 12:22 |
| *** weshay is now known as weshay_pto | 12:22 | |
| rcarrillocruz | k, got to copy paste stuff to the one crinkle shared | 12:22 |
| rcarrillocruz | i made a copy of it for my drafting | 12:22 |
| *** dansmith has quit IRC | 12:23 | |
| AJaeger | pabelanger: enjoy! | 12:23 |
| *** tiswanso has joined #openstack-infra | 12:23 | |
| *** kgiusti has joined #openstack-infra | 12:24 | |
| *** amitgandhinz has joined #openstack-infra | 12:25 | |
| *** trown|outtypewww is now known as trown | 12:25 | |
| crinkle | pabelanger: rcarrillocruz cool | 12:25 |
| *** EricGonczer_ has quit IRC | 12:25 | |
| *** tiswanso_ has quit IRC | 12:26 | |
| rcarrillocruz | i think we should meet on monday to sync up and prepare it | 12:26 |
| rcarrillocruz | crinkle, pabelanger ^ | 12:26 |
| rcarrillocruz | ? | 12:26 |
| pabelanger | rcarrillocruz: crinkle: that works for me, assuming everybody here on Monday | 12:27 |
| pabelanger | otherwise, we could get into pbx.o.o again tomorrow | 12:28 |
| crinkle | pabelanger: rcarrillocruz i won't be in till late on monday | 12:28 |
| pabelanger | k, how about we sync up first thing tomorrow? | 12:29 |
| crinkle | works for me | 12:29 |
| rcarrillocruz | k | 12:29 |
| pabelanger | I'm in UTC+2 right now FYI | 12:30 |
| *** abregman has joined #openstack-infra | 12:33 | |
| *** tiswanso has quit IRC | 12:34 | |
| *** Jeffrey4l has quit IRC | 12:35 | |
| frickler | EmilienM: pabelanger: nibalizer: I made a fix for https://review.openstack.org/253616 , please check whether I did the right thing, at least it passes jenkins now. would be great if we could this mirror running, there are lots of rubygem timeouts on chef jobs | 12:36 |
| EmilienM | frickler: oh nice, thanks for helping on this | 12:37 |
| *** amitgandhinz has quit IRC | 12:37 | |
| *** thorst_ has quit IRC | 12:37 | |
| pabelanger | frickler: EmilienM: I hope to spend some time at the summit working on this | 12:38 |
| EmilienM | pabelanger: thanks, I would be happy to helpo | 12:39 |
| EmilienM | help | 12:39 |
| *** rlandy has joined #openstack-infra | 12:42 | |
| *** ccamacho|lunch is now known as ccamacho | 12:42 | |
| pabelanger | EmilienM: frickler: -1, easy fix. Once that is updated, I think we can land the patch | 12:44 |
| *** tnovacik has joined #openstack-infra | 12:44 | |
| *** zeih has quit IRC | 12:45 | |
| *** tlian has joined #openstack-infra | 12:45 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Mistral to scenario003 https://review.openstack.org/368805 | 12:45 |
| *** zeih has joined #openstack-infra | 12:46 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 12:46 |
| frickler | pabelanger: EmilienM: ^^ | 12:46 |
| EmilienM | frickler: thx | 12:46 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 12:48 |
| *** jpena|lunch is now known as jpena | 12:48 | |
| *** gordc has joined #openstack-infra | 12:49 | |
| *** derekh has quit IRC | 12:50 | |
| *** mtanino has quit IRC | 12:51 | |
| *** zeih has quit IRC | 12:51 | |
| *** rhallisey has joined #openstack-infra | 12:51 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: Add new openssh-server element https://review.openstack.org/389171 | 12:52 |
| *** rcernin has joined #openstack-infra | 12:53 | |
| *** jcoufal has joined #openstack-infra | 12:55 | |
| *** xavierr has joined #openstack-infra | 12:56 | |
| *** pgadiya has quit IRC | 12:56 | |
| xavierr | good morning Infra | 12:57 |
| *** esikachev has joined #openstack-infra | 12:58 | |
| xavierr | hey, I uploaded a new tag to python-oneviewclient and like always it should be upload automatically to pypi, however it was not. any ideas? | 12:59 |
| xavierr | http://logs.openstack.org/c5/c5dc97fadf24ac6ddf7075f3dbc6456878f8ee56/release/ | 13:01 |
| *** jaosorior is now known as jaosorior_brb | 13:01 | |
| robcresswell | xavierr: IIRC new tags aren't actually automatically processed. There's a next step that has to be triggered. Also this is a question for openstack-release, not openstack-infra, isnt it? | 13:01 |
| *** bin_ has joined #openstack-infra | 13:02 | |
| *** gmann__ has quit IRC | 13:02 | |
| *** esikachev has quit IRC | 13:03 | |
| *** vikrant|brb has quit IRC | 13:03 | |
| frickler | so if I have a grenade error from zl01, should I just recheck or will than one be fixed soon, too? | 13:04 |
| xavierr | robcresswell: people from Ironic told me to ask here, but ty anyways | 13:04 |
| *** vsaienko has joined #openstack-infra | 13:04 | |
| mordred | yah. I just removed zl01 from the emergency file - will get it updated real quickly | 13:05 |
| mordred | pabelanger, robcresswell: morning | 13:05 |
| frickler | mordred: cool, thx | 13:05 |
| robcresswell | xavierr: I think you might have more luck asking in openstack-release :) | 13:05 |
| robcresswell | mordred: \o | 13:05 |
| *** Goneri has joined #openstack-infra | 13:07 | |
| AJaeger | xavierr: right now something is broken, please wait with tagging until it's fixed - see http://lists.openstack.org/pipermail/openstack-dev/2016-October/106144.html | 13:08 |
| *** jistr is now known as jistr|biab | 13:08 | |
| *** mkoderer has joined #openstack-infra | 13:08 | |
| *** derekh has joined #openstack-infra | 13:09 | |
| *** sree has joined #openstack-infra | 13:09 | |
| pabelanger | mordred: o/ | 13:10 |
| pabelanger | xavierr: I'll take a look | 13:11 |
| pabelanger | http://logs.openstack.org/c5/c5dc97fadf24ac6ddf7075f3dbc6456878f8ee56/release/python-oneviewclient-pypi-both-upload/e5bbede/console.html is the reason | 13:11 |
| *** jistr|biab is now known as jistr | 13:11 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** adrian_otto has joined #openstack-infra | 13:14 | |
| xavierr | pabelanger: thanks, but there is anything I can do, or only guys from release can? :) | 13:15 |
| pabelanger | xavierr: trying to find out why it failed | 13:18 |
| pabelanger | it looks like the job timed out | 13:19 |
| pabelanger | mordred: have we seen this yet? http://logs.openstack.org/c5/c5dc97fadf24ac6ddf7075f3dbc6456878f8ee56/release/python-oneviewclient-pypi-both-upload/e5bbede/_zuul_ansible/ansible_log.txt | 13:19 |
| pabelanger | looks like ansible could finish the file task | 13:20 |
| *** mriedem has joined #openstack-infra | 13:20 | |
| *** mdrabe has joined #openstack-infra | 13:20 | |
| pabelanger | actually, the command | 13:21 |
| *** sdake has joined #openstack-infra | 13:21 | |
| AJaeger | xavierr: nothing you can do right now, just wait. Once the underlying problem is fixed, we can discuss what to do - we might be able to reenqueu the job... | 13:21 |
| mordred | pabelanger: that's weird | 13:21 |
| pabelanger | looks like the command ran, cause there is output in console.html | 13:22 |
| pabelanger | but didn't exit properly | 13:22 |
| *** nicolasbock has quit IRC | 13:23 | |
| xavierr | AJaeger: ok, I'll wait until the end of this day :) | 13:23 |
| xavierr | thank you infra! | 13:23 |
| *** florianf has quit IRC | 13:24 | |
| *** xavierr has left #openstack-infra | 13:25 | |
| *** nicolasbock has joined #openstack-infra | 13:27 | |
| njohnston | I get "Code Review - Error 500 Internal server error" when I try and cherry pick https://review.openstack.org/#/c/387398/ to stable/newton. I had another person try and they got the same issue. Does anyone know what could be causing this problem? | 13:27 |
| *** matt-borland has joined #openstack-infra | 13:27 | |
| *** cardeois has joined #openstack-infra | 13:27 | |
| *** rfolco has joined #openstack-infra | 13:30 | |
| *** zhurong has quit IRC | 13:30 | |
| *** vsaienko has quit IRC | 13:30 | |
| *** dizquierdo has joined #openstack-infra | 13:31 | |
| frickler | njohnston: hmm, that looks strange, I'm getting that error too, but a local cherry-pick works just fine. I guess someone might have to check the logs on gerrit | 13:32 |
| dhellmann | robcresswell, xavierr: we're seeing errors on the signing node with the jobs that run when a tag is being processed http://logs.openstack.org/8f/8f9f8de5a6177eb07ba067e78fade6c9ccde5df1/release-post/tag-releases/1823078/console.html | 13:32 |
| dhellmann | oh, AJaeger beat me to it ;-) | 13:33 |
| *** billiebobthorty has joined #openstack-infra | 13:33 | |
| *** kjackal_ has joined #openstack-infra | 13:33 | |
| *** sdake has quit IRC | 13:33 | |
| njohnston | Thanks for looking, frickler | 13:35 |
| kjackal_ | Hi there, I get a "Cannot store contact information" so I cannot push anything for review. Any idea why? | 13:37 |
| *** zhurong has joined #openstack-infra | 13:37 | |
| *** yamahata has joined #openstack-infra | 13:37 | |
| mordred | pabelanger: I have restarted zl01 - it was in the emergency file last night so didn't get puppet updates to get the git repo updated | 13:38 |
| *** amitgandhinz has joined #openstack-infra | 13:38 | |
| mordred | pabelanger: I've also removed it from the emergency file, so it should be back in the fleet properly | 13:38 |
| *** florianf has joined #openstack-infra | 13:38 | |
| AJaeger | kjackal_: This needs your gerrit preferred e-mail address to match a primary e-mail address for a foundation individual member account. | 13:38 |
| *** nicolasbock has quit IRC | 13:39 | |
| kjackal_ | AJaeger: thank you, let me try to parse that. | 13:39 |
| AJaeger | kjackal_: If you already followed the instructions (all, in order!) at http://docs.openstack.org/infra/manual/developers.html#account-setup and still get that, see https://ask.openstack.org/question/56720 for additional troubleshooting tips. | 13:39 |
| *** hurgleburgler has joined #openstack-infra | 13:39 | |
| wznoinsk | hi infra, would someone have a moment for trivial gerrit group add review https://review.openstack.org/#/c/388894/ ? | 13:42 |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| wznoinsk | thanks rcarrillocruz | 13:45 |
| rcarrillocruz | np | 13:45 |
| *** EricGonczer_ has joined #openstack-infra | 13:46 | |
| openstackgerrit | Merged openstack-infra/project-config: gerrit: add intel-nfv-ci-tests-ci group https://review.openstack.org/388894 | 13:48 |
| *** inc0 has joined #openstack-infra | 13:50 | |
| *** sree_ has joined #openstack-infra | 13:51 | |
| *** jheroux has joined #openstack-infra | 13:51 | |
| dhellmann | pabelanger : is there anything to report on those signing job failures? it seems weird for it to be stuck on a call to "rm" like that. | 13:51 |
| *** sree_ is now known as Guest49607 | 13:51 | |
| *** sdague has joined #openstack-infra | 13:51 | |
| *** sree has quit IRC | 13:53 | |
| *** vsaienko has joined #openstack-infra | 13:54 | |
| *** rfolco has quit IRC | 13:54 | |
| pabelanger | dhellmann: not yet, still looking. We're executing a new code path, now that we removed async support. I suspect the rm command is successful, but ansible didn't parse the return code properly | 13:55 |
| dhellmann | pabelanger : ah, ok | 13:56 |
| dhellmann | we're compiling a list of the tag jobs that will need to be re-run in https://etherpad.openstack.org/p/6mZZeAigiR | 13:57 |
| pabelanger | good idea, I'm sure there as been more failures | 13:57 |
| *** makowals has quit IRC | 13:57 | |
| * dhellmann nods | 13:58 | |
| *** makowals has joined #openstack-infra | 13:59 | |
| openstackgerrit | Anshul Jain proposed openstack/diskimage-builder: dib element to create customized image with cinder local attach-detach functionality. https://review.openstack.org/385880 | 14:00 |
| *** esikachev has joined #openstack-infra | 14:01 | |
| *** spzala has joined #openstack-infra | 14:01 | |
| mordred | pabelanger: AJaeger was saying something about sudo config being messed up on zlstatic perhaps | 14:01 |
| *** nicolasbock has joined #openstack-infra | 14:01 | |
| AJaeger | mordred: not me - that was in the email I referenced | 14:02 |
| pabelanger | Oh | 14:02 |
| pabelanger | AJaeger: which email was that? | 14:02 |
| *** EricGonczer_ has quit IRC | 14:02 | |
| *** adrian_otto has quit IRC | 14:02 | |
| mordred | pabelanger: oh! are our requiretty settings not set up properly? | 14:03 |
| *** sdague has quit IRC | 14:03 | |
| pabelanger | mordred: let me check | 14:03 |
| mordred | pabelanger: there's something about that relatde to the pipelining change - and I thought we'd verified our settings we correct, but maybe they aren't? | 14:03 |
| *** yamamoto has joined #openstack-infra | 14:04 | |
| *** jaosorior_brb is now known as jaosorior | 14:05 | |
| *** EricGonczer_ has joined #openstack-infra | 14:05 | |
| *** EricGonczer_ has quit IRC | 14:06 | |
| *** EricGonczer_ has joined #openstack-infra | 14:06 | |
| AJaeger | mordred: see http://lists.openstack.org/pipermail/openstack-dev/2016-October/106144.html | 14:06 |
| mordred | AJaeger: nod | 14:07 |
| kashyap | clarkb: fungi: Hi, just curious -- wonder is it possible to enable KVM nested virt (Intel / AMD) on the Kernels on Gate host? Perhaps for "limited machines"? | 14:08 |
| *** ijw has joined #openstack-infra | 14:08 | |
| mordred | kashyap: it is not, sorry | 14:08 |
| kashyap | mordred: I realize, you have to elaborate more than that... | 14:08 |
| kashyap | mordred: Is the fear that it "breaks the world"? | 14:08 |
| kashyap | That distro kernels haven't enabled it? Stability concern? | 14:08 |
| mordred | well, that's part of it - and it's not fear, the times in the past it's been enabled it has in fact broken the world | 14:09 |
| mordred | but more importantly - we do not run the clouds we use for the gate | 14:09 |
| mordred | and it requires enablement at the cloud provider level | 14:09 |
| mordred | most of our clouds do not have it enabled - and some cannot provide it in the first place | 14:09 |
| pabelanger | mordred: requiretty looks good, but we could add Defaults:jenkins !requiretty | 14:09 |
| pabelanger | to be safe in to our sudoers.d file | 14:09 |
| mordred | pabelanger: darn. I was hoping it wouldthat | 14:09 |
| kashyap | mordred: If it's security. FWIW, at the recent KVMForum in Toronto, security engineers (IIRC from Google) have talked about audit of the nested KVM code...and haven't found anything glaring | 14:10 |
| *** xarses has quit IRC | 14:10 | |
| *** sdague has joined #openstack-infra | 14:10 | |
| kashyap | mordred: Ah, reading your other comments | 14:10 |
| mordred | kashyap: and it wasn't really security we were concerned about - as much as the times it has been tried when clouds have enabled it, it has been much more unstable and jobs have failed for extremely hard to debug / weird reasons | 14:11 |
| kashyap | mordred: Hmm, I don't know when were these "times in the past past". But upstream there have been improvements consistently. | 14:12 |
| mordred | but that, combined with the fact that we simply don't have the control over our clouds to make such a choice anyway | 14:12 |
| kashyap | mordred: Right, that's a fair point | 14:12 |
| kashyap | I think part of them are on Rackspace, which run Xen | 14:12 |
| mordred | yah. | 14:12 |
| *** amitgandhinz has quit IRC | 14:12 | |
| mordred | nested kvm virt there - much harder to get :) | 14:12 |
| kashyap | Therefore, we're stuck in limbo. | 14:12 |
| mordred | yah. there is not a good nested virt story we have at our disposal currently | 14:13 |
| *** yamamoto has quit IRC | 14:13 | |
| *** ijw has quit IRC | 14:13 | |
| kashyap | mordred: The point to consider is (credit where its' due: dansmith raised it some week ago) | 14:13 |
| kashyap | We use plain emulation (QEMU "TCG") through out in the Gate for testing. However, that same configuration is are not "recommended" for operators (for performance reasons) to run what we test in the Gate | 14:15 |
| *** rbrndt has joined #openstack-infra | 14:16 | |
| kashyap | Anyhow...Thanks for the comment. | 14:16 |
| mordred | kashyap: totally! I wish we had a better option to address that | 14:17 |
| mordred | pabelanger: I have reproduced the hang | 14:18 |
| pabelanger | mordred: does it have to do with sudo asking for a password? | 14:18 |
| *** mtanino has joined #openstack-infra | 14:19 | |
| mordred | pabelanger: that I don't know yet - but I have a _very_ simple playbook that is exhibiting it | 14:19 |
| pabelanger | k | 14:19 |
| pabelanger | when I run sudo rm -f /etc/sudoers.d/jenkins-sudo I'm prompted for password as jenkins (expect) | 14:19 |
| *** abregman is now known as abregman|afk | 14:19 | |
| pabelanger | wasn't sure if command would handle that | 14:19 |
| openstackgerrit | Merged openstack-infra/shade: Add test for os_keystone_domain Ansible module https://review.openstack.org/388697 | 14:20 |
| openstackgerrit | Merged openstack-infra/shade: Allow setting env variables for functional options https://review.openstack.org/388983 | 14:20 |
| pabelanger | 2016-09-26 17:31:46.481247 | sudo: no tty present and no askpass program specified | 14:20 |
| pabelanger | is what we'd get before | 14:20 |
| mordred | pabelanger: I have removed our command module and the hang still happens | 14:21 |
| fungi | pabelanger: mordred: sounds like it may be related to a change in how ansible is executing the job? | 14:21 |
| fungi | just now catching up, my workout ran a lot longer than expected this morning | 14:21 |
| pabelanger | mordred: ack | 14:22 |
| mordred | pabelanger: ok. I'm not sure what's going on - but a playbook from zlstatic to proposal.slave hangs if I run sudo in a shell command | 14:23 |
| mordred | it does not hang if I do not | 14:23 |
| openstackgerrit | Merged openstack-infra/zuul: Add back timeout_var logic https://review.openstack.org/389108 | 14:23 |
| fungi | as a quick fix we could just remove the revoke-sudo builder from jobs that run on static job nodes | 14:23 |
| fungi | it's not needed, it's just there for consistency | 14:24 |
| pabelanger | mordred: okay, that is what I was thinking. I wonder if the becomes logic is coming into play | 14:24 |
| mordred | pabelanger: I'm not running with becomes | 14:24 |
| pabelanger | since you'd not run command: sudo foo | 14:24 |
| mordred | pabelanger: well, we do in the revoke-sudo builder | 14:24 |
| pabelanger | youd do command foo | 14:24 |
| pabelanger | become: yes | 14:24 |
| mordred | because we aren't actually writing ansible | 14:24 |
| fungi | alternatively, we could tweak revoke-sudo to (somehow) drop the tty for stdin... maybe </dev/null or something | 14:24 |
| pabelanger | ya | 14:24 |
| mordred | we're translating from jjb to generated ansible | 14:24 |
| pabelanger | I've never done sudo with ansible that way, I'd have to test | 14:25 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-bandersnatch: Fix bandersnatch crons to support full sync https://review.openstack.org/383836 | 14:25 |
| mordred | fungi: just tried that - it doesn't help | 14:25 |
| * fungi grumbles at computers | 14:25 | |
| jeblair | there's lots of sudo commands we run -- does this happen with all of them or just this one? | 14:26 |
| mordred | jeblair: this is the only one we know about | 14:26 |
| fungi | just this one, because jenkins lacks sudo perms | 14:26 |
| pabelanger | Ya, I think it is only an issue when we are prompted for a password | 14:26 |
| *** Guest49607 has quit IRC | 14:26 | |
| fungi | i mean, it conceivably happens elsewhere if a job tries to sudo when it shouldn't, but we generally fail those anyway so they're hopefully rare | 14:26 |
| jeblair | anyone have a link to a failed run handy? | 14:27 |
| pabelanger | jeblair: http://logs.openstack.org/c5/c5dc97fadf24ac6ddf7075f3dbc6456878f8ee56/release/python-oneviewclient-pypi-both-upload/e5bbede/console.html | 14:27 |
| jeblair | thx | 14:27 |
| fungi | we could add -A /bin/false maybe? | 14:27 |
| mordred | jeblair: I have a copy of a setup in root@zlstatic01:~/tmpK08vMM | 14:27 |
| jeblair | we also do this: | 14:28 |
| *** rossella_s has quit IRC | 14:28 | |
| jeblair | ! sudo -n true | 14:28 |
| openstack | jeblair: Error: "sudo" is not a valid command. | 14:28 |
| jeblair | gr | 14:28 |
| AJaeger | ;) | 14:28 |
| *** lucasagomes is now known as lucas-hungry | 14:28 | |
| jeblair | at the end of the revoke-sudo command | 14:28 |
| *** zhurong has quit IRC | 14:28 | |
| fungi | sudo -n true doesn't prompt | 14:28 |
| *** rossella_s has joined #openstack-infra | 14:28 | |
| fungi | ahh, right, we could add -n instead of -A /bin/false | 14:28 |
| mordred | jeblair: and a playbook called test_playbook | 14:28 |
| jeblair | fungi: ah, the -n? | 14:28 |
| pabelanger | ya | 14:28 |
| jeblair | mordred: thx | 14:29 |
| fungi | i mean, we likely _always_ want sudo -n in our jobs anyway. there is nobody/nothing around to interact with it ever | 14:29 |
| pabelanger | $ sudo -n rm /etc/sudoers.d/jenkins-sudo | 14:29 |
| pabelanger | sudo: a password is required | 14:29 |
| pabelanger | fungi: agreed | 14:29 |
| *** xarses has joined #openstack-infra | 14:30 | |
| jeblair | fungi: right, but also, we can't guard against that perfectly (just like we failed to guard against adding revoke-sudo to jobs that don't need it) | 14:30 |
| mordred | I can verify that adding -n does not cause it to hang | 14:30 |
| cardeois | pabelanger I see in the history that you were talking about async support that was removed. I have some jobs failing since yesterday that seems to be related to that. Can you elaborate or point me to more info? | 14:30 |
| mordred | let me put the new console module and pipelining back in place | 14:30 |
| cardeois | (My build failing http://logs.openstack.org/40/388940/1/check/gate-js-openstack-lib-nodejs4-npm-run-lint/fa9cc5e/console.html#_2016-10-20_06_13_59_080547) | 14:30 |
| mordred | cardeois: we may be just about done figuring it out - hang on just a little bit ... | 14:30 |
| cardeois | alright thanks | 14:31 |
| pabelanger | cardeois: Ya, we are working on them as they come up | 14:31 |
| pabelanger | cardeois: let me look at the log | 14:31 |
| mordred | jeblair: ok - doing sudo -n instead of plan sudo works | 14:31 |
| mordred | I mean, it fails the task - but it does not hang | 14:31 |
| *** knangia has joined #openstack-infra | 14:31 | |
| mordred | jeblair: and that's with our command module in place and with pipelining turned on (neither of those seem to actually be related) | 14:32 |
| cardeois | pabelanger sure thanks. It seems related to xfvb we run in background in order to launch a chrome or firefox later for JS tests | 14:32 |
| fungi | we could also inject SUDO_ASKPASS=/bin/false into the calling environment maybe? though that won't make it through tox or user switching like devstack does | 14:32 |
| jeblair | yeah. i agree this is an immediate solution to getting signing jobs to run. but i don't think we can leave zuul in this state. | 14:32 |
| pabelanger | mordred: since you can test, what happens when you remove -n and add become: yes to the task? Does ansible raise an exception? | 14:33 |
| pabelanger | cardeois: Ya, we haven't see that one yet. | 14:33 |
| jeblair | cardeois: we beleieve the problem exhibited in that job has been fixed, can you recheck it? | 14:33 |
| mordred | fungi: SUDO_ASKPASS in the environment did not work | 14:33 |
| jeblair | pabelanger: that was the close() problem | 14:34 |
| cardeois | jetblair will do | 14:34 |
| pabelanger | jeblair: Ah, thank you. I missed that one from yesterday | 14:34 |
| rcarrillocruz | crinkle: did you have a sequence diagram about bifrost provisioning at all? | 14:34 |
| mordred | pabelanger: become: True fails and does not hang | 14:34 |
| rcarrillocruz | or maybe i'm confused and i had it in on of my old tech talks i gave about it | 14:34 |
| * rcarrillocruz confused | 14:34 | |
| *** hongbin has joined #openstack-infra | 14:35 | |
| fungi | mordred: indeed, the sudo manpage indicates SUDO_ASKPASS is honored, but testing confirms it doesn't seem to help | 14:35 |
| *** Jeffrey4l has joined #openstack-infra | 14:35 | |
| mordred | but I agree, we don't want things to hang when someone uses sudo when they're not supposed to - we want those things to fail | 14:35 |
| pabelanger | mordred: okay, that is what I would expect. | 14:35 |
| *** sputnik13 has joined #openstack-infra | 14:35 | |
| fungi | also obviously attacking the symptom rather than the root cause, but we could replace sudo with a wrapper | 14:36 |
| *** nherciu has quit IRC | 14:36 | |
| AJaeger | jeblair: do you have time to fix the zuul-merger problem with the periodic translation jobs before the OpenStack summit? Or should I revert my change that exposed it for now? | 14:37 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Adjust functional test filter for OpenStack-Ansible https://review.openstack.org/389232 | 14:38 |
| crinkle | rcarrillocruz: i don't think i have anything specific to bifrost, the diagrams i have a little more big-picture | 14:38 |
| jeblair | AJaeger: i'd like to get to it today, but i can't promise i will; if you need to revert, that's fine. | 14:38 |
| rcarrillocruz | yeah, i would have sworn i had something, i'll check my messy hard drive for old presentations, it could be i did myself | 14:39 |
| jeblair | mordred: is the zuul command module in place in your tmpdir or no? | 14:39 |
| AJaeger | jeblair: I can wait another day | 14:39 |
| mordred | jeblair: it is - and I'm poking at it right now - you might want to make a copy of that dir if you want to poke too | 14:43 |
| *** tpsilva has joined #openstack-infra | 14:43 | |
| *** ssbarnea has quit IRC | 14:44 | |
| AJaeger | jeblair: thanks | 14:45 |
| *** makowals has quit IRC | 14:45 | |
| *** amotoki has joined #openstack-infra | 14:45 | |
| pabelanger | jeblair: mordred: should I start rolling restarts on zuul-launchers to pick up devstack-gate fix? Or hold off for some more potential fixes | 14:45 |
| jeblair | pabelanger: i wouldn't do a rolling restart anyway, it would take all day and we're certain to want another. | 14:49 |
| jeblair | pabelanger: i'd say hold off for a few more mins | 14:49 |
| jeblair | (but if we did want to restart, do a hard restart) | 14:49 |
| pabelanger | okay, hard works too. | 14:50 |
| *** oanson has joined #openstack-infra | 14:50 | |
| pabelanger | in that case, happy to wait | 14:50 |
| fungi | what was the recent ansible change? i saw something about enabling pipelining... | 14:53 |
| *** flepied has quit IRC | 14:53 | |
| pabelanger | fungi: ya, that was enabled | 14:53 |
| pabelanger | plus we removed zuul_runner and replaced it with a modified version of command | 14:53 |
| *** spzala has quit IRC | 14:53 | |
| fungi | it looks like that utilizes the remote interpreter's stdin, so i can sort of see how it might change behavior this way | 14:53 |
| fungi | pipelining i mean | 14:54 |
| mordred | pipelining does not affect this | 14:54 |
| mordred | I have tested with it on and off | 14:55 |
| fungi | got it. i saw you mention reenabling pipelining above, but wasn't sure what the outcome had been of testing without | 14:56 |
| fungi | so suspicion at this point is that it has to do with the command module? | 14:56 |
| *** vsaienko has quit IRC | 14:56 | |
| *** nicolasbock has quit IRC | 14:56 | |
| mordred | yah - although not necessarily with our version - the problem manifests both with and without our version of the command module in place | 14:57 |
| mordred | I have just verified that running the task under async fails the command properly | 14:57 |
| mordred | jeblair: ^^ | 14:57 |
| jeblair | ack | 14:57 |
| *** marst has quit IRC | 14:58 | |
| *** nicolasbock has joined #openstack-infra | 14:58 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on same event multiple times on schedule fixed code to prevent that situation https://review.openstack.org/389239 | 15:00 |
| fungi | it looks like we could maybe do something like authenticate=no in sudoers, or add a greedy glob that sets NOPASSWD | 15:02 |
| *** amotoki has quit IRC | 15:02 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on same event multiple times on schedule fixed code to prevent that situation https://review.openstack.org/389239 | 15:02 |
| *** flepied has joined #openstack-infra | 15:02 | |
| fungi | that might be an effective way to guard against similar situations in the future (but not immediately as it would need new images) | 15:02 |
| *** esikache1 has joined #openstack-infra | 15:02 | |
| *** r-daneel has joined #openstack-infra | 15:04 | |
| *** esikachev has quit IRC | 15:04 | |
| *** makowals has joined #openstack-infra | 15:04 | |
| jeblair | fungi: fundamentally, this is something that needs to be fixed in zuul/ansible. we can't have a system that lets people shoot themselves in the foot like this. | 15:05 |
| fungi | agreed | 15:05 |
| *** amotoki has joined #openstack-infra | 15:05 | |
| *** piet_ has joined #openstack-infra | 15:05 | |
| fungi | but sounds like there's a good chance it may be a behavior-changing patch to ansible (or new option) | 15:05 |
| mordred | jeblair: fwiw, I have been trying various absurd things in the command module to try to get python to make the subprocess invocation happy | 15:05 |
| mordred | so far I have not been successful | 15:06 |
| mnaser | just saw a failure of a job: "W: The repository 'http://mirror.mtl01.internap.openstack.org/ubuntu xenial Release' is not signed." | 15:06 |
| mnaser | http://logs.openstack.org/90/389190/1/check/gate-tempest-dsvm-neutron-src-oslo.messaging-newton/04cc851/console.html | 15:06 |
| *** eharney has quit IRC | 15:06 | |
| jeblair | mordred: yeah, i'm about to start doing that :/ | 15:06 |
| fungi | mnaser: that shouldn't be a failure, just a warning | 15:06 |
| mnaser | python-yaml installation failed because it could not be authenticated | 15:06 |
| mordred | jeblair: my most recent forray has been poking at the pty module | 15:06 |
| mnaser | ooo | 15:06 |
| mnaser | you're right | 15:06 |
| pabelanger | mnaser: that not signed is expected. We don't actually have gpg signed repos for debuntu | 15:06 |
| *** sdague has quit IRC | 15:06 | |
| pabelanger | just seen this linked in #tripleo: 2016-10-20 14:54:20,868 p=16333 u=zuul | fatal: [node]: FAILED! => {"changed": false, "cmd": "/tmp/04-5281fb81a79b4d9f99cac09f3108924d.sh", "failed": true, "msg": "close() called during concurrent operation on the same file object.", "rc": null} | 15:08 |
| pabelanger | have we seen that before? | 15:08 |
| jeblair | pabelanger: that's the close() bug that was fixed earlier | 15:08 |
| *** amitgandhinz has joined #openstack-infra | 15:08 | |
| pabelanger | jeblair: okay, do we need to restart launchers? | 15:08 |
| jeblair | pabelanger: that timestamp is pretty recent though | 15:08 |
| pabelanger | ya | 15:09 |
| pabelanger | it just happened | 15:09 |
| jeblair | pabelanger: i gathered from irc logs that was done. it may be worth tracking down which zuul launcher that ran on and see if it was actually restarted with the change in place. | 15:09 |
| pabelanger | okay, I can do that | 15:09 |
| *** marst has joined #openstack-infra | 15:10 | |
| *** sdague has joined #openstack-infra | 15:10 | |
| pabelanger | looks like zl01, checking now | 15:10 |
| *** mdrabe has quit IRC | 15:10 | |
| mordred | I restarted zl01 this morning with the change in place | 15:10 |
| pabelanger | k | 15:10 |
| *** nicolasbock has quit IRC | 15:11 | |
| mordred | at 13:38:08 | 15:11 |
| *** thorst_ has joined #openstack-infra | 15:11 | |
| fungi | does the shell module suffer the same problem? or are there reasons we shouldn't use that one? | 15:11 |
| *** thorst_ has quit IRC | 15:11 | |
| mordred | fungi: it's the same module | 15:11 |
| jeblair | fungi: it's actuall that | 15:11 |
| fungi | ahh | 15:11 |
| fungi | the ansible docs make it sound like the command module and the shell module are distinct | 15:12 |
| mordred | well, basically, using the shell module causes a parameter to be set | 15:12 |
| *** thorst_ has joined #openstack-infra | 15:13 | |
| mordred | so if you use shell, it tells python.subprocess to spawn a subshell, if you use command, it does not | 15:13 |
| clarkb | fungi: I think they were for a long time but were collapsed in 2.0 (or other relatively recent version) | 15:13 |
| *** yamamoto has joined #openstack-infra | 15:14 | |
| *** yamamoto has quit IRC | 15:14 | |
| *** yamamoto has joined #openstack-infra | 15:15 | |
| rcarrillocruz | in the code, they're pretty much the same | 15:16 |
| rcarrillocruz | as mordred says, it pretty much differ on a param saying 'uses_shell=True' | 15:16 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Fix on PHP 503 error updated pm.max_children from 300 to 400 to avoid 503 http error. https://review.openstack.org/389244 | 15:17 |
| jeblair | mordred: zuul_runner does not pass stdin to its command, but command module does | 15:17 |
| jeblair | #stdin=st_in, | 15:17 |
| *** mdrabe has joined #openstack-infra | 15:18 | |
| jeblair | will 'fix' the command module | 15:18 |
| mordred | oh - really? | 15:18 |
| mordred | jeblair: it still hangs for me | 15:18 |
| jeblair | hrm, let me make sure i didn't contaminate my test | 15:18 |
| mordred | jeblair: st_in defaults to None, fwiw. it's only set to something if someone passes in "data" as a parameter to the command module | 15:19 |
| jeblair | mordred: yep, bad test, sorry | 15:19 |
| *** jcoufal_ has joined #openstack-infra | 15:19 | |
| mordred | darn. I was hoping I'd screwed up mine | 15:19 |
| *** spzala has joined #openstack-infra | 15:20 | |
| dhellmann | jeblair , mordred : you're not running under python 3 by any chance, are you? there were some changes to the way subprocess starts new processes under py3. | 15:21 |
| *** jcoufal has quit IRC | 15:21 | |
| mordred | dhellmann: we are not | 15:22 |
| dhellmann | ok, good | 15:22 |
| *** eharney has joined #openstack-infra | 15:22 | |
| mordred | the ansible folks just this last cycle started making things py3 capable | 15:22 |
| dhellmann | I think those changes were just related to signal handling, but it doesn't matter | 15:22 |
| *** kairat has left #openstack-infra | 15:22 | |
| *** yamamoto has quit IRC | 15:23 | |
| jeblair | mordred: i'm leaning towards thinking this is a side effect of async | 15:24 |
| jeblair | i think zuul_runner suffers this as well | 15:24 |
| *** nicolasbock has joined #openstack-infra | 15:24 | |
| *** amotoki has quit IRC | 15:24 | |
| *** esikache1 has quit IRC | 15:25 | |
| *** yamahata has quit IRC | 15:25 | |
| mordred | jeblair: and we just didn't notice because we were running under async which was running in a whole other daemon subprocess? | 15:25 |
| jeblair | yep | 15:26 |
| mordred | yah. I agree | 15:26 |
| *** amotoki has joined #openstack-infra | 15:26 | |
| *** panda is now known as panda|bbl | 15:26 | |
| pabelanger | jeblair: mordred: Ya, we are still running the unfixed plugin in zl01. We need another restart | 15:26 |
| mordred | pabelanger: sigh | 15:26 |
| jeblair | pabelanger, mordred: maybe pip install failed again :/ | 15:27 |
| pabelanger | puppet updated zuul about 5mins after you restarted | 15:27 |
| pabelanger | jeblair: possible, I don't see puppet kicking off a zuul_install exec until build-var patch landed on disk | 15:28 |
| jeblair | pabelanger: ah, then we may have just suffered from the puppetmaster oom | 15:28 |
| *** sbadia has quit IRC | 15:28 | |
| jeblair | mordred: i made a simple copy of zuul_runner and it has the problem: http://paste.openstack.org/show/586594/ | 15:29 |
| jeblair | also, trying 'stdin=PIPE' and 'proc.stdin.close()' in that doesn't help | 15:29 |
| pabelanger | jeblair: Oh, nice (well no). Didn't know puppet would be left in a broken state on the far end | 15:29 |
| *** eharney has quit IRC | 15:30 | |
| jeblair | pabelanger: er, i don't know about broken -- it sounds like you're saying it just didn't get around to running for a while | 15:30 |
| jeblair | like, 6 hours | 15:30 |
| pabelanger | jeblair: Oh, i see. Ya, that makes more sense. | 15:31 |
| pabelanger | Also, I confused project-config update with zuul | 15:31 |
| *** lucas-hungry is now known as lucasagomes | 15:32 | |
| *** vhosakot has joined #openstack-infra | 15:32 | |
| *** makowals has quit IRC | 15:33 | |
| *** dizquierdo has quit IRC | 15:33 | |
| *** sbadia has joined #openstack-infra | 15:33 | |
| *** amotoki has quit IRC | 15:34 | |
| *** baoli_ has quit IRC | 15:35 | |
| *** sputnik13 has quit IRC | 15:36 | |
| *** spzala has quit IRC | 15:37 | |
| *** billiebobthorty has quit IRC | 15:38 | |
| *** priteau has joined #openstack-infra | 15:39 | |
| *** nicolasbock has quit IRC | 15:39 | |
| *** yolanda has joined #openstack-infra | 15:39 | |
| *** andreas_s has quit IRC | 15:42 | |
| *** eharney has joined #openstack-infra | 15:43 | |
| *** amitgandhinz has quit IRC | 15:43 | |
| mordred | jeblair: woot! I made something which does not hang | 15:43 |
| jeblair | mordred: neat! whadyado? | 15:44 |
| fungi | oh?!? | 15:44 |
| mordred | jeblair: I lost the process return code in the process, so I need to figure that out now :) | 15:44 |
| mordred | jeblair: I used pty.spawn instead of subprocess.Popen + Thread | 15:44 |
| *** jaosorior has quit IRC | 15:44 | |
| jeblair | mordred: ah | 15:45 |
| mordred | jeblair: amusingly enough, pty.spawn has the same semantics as our follow process - or basically takes a function that works just like follow | 15:45 |
| jeblair | i was trying to hook it up to a pty and not making headway | 15:45 |
| mordred | jeblair: oo! I got the return code back | 15:46 |
| mordred | that was easy | 15:46 |
| *** spzala has joined #openstack-infra | 15:46 | |
| mordred | jeblair: let me copy what I've got somewhere so you can look at it and we can clean it up | 15:46 |
| jeblair | kk | 15:46 |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| *** sflanigan has quit IRC | 15:49 | |
| *** spzala has quit IRC | 15:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use pty.spawn to spawn the subprocess https://review.openstack.org/389260 | 15:51 |
| mordred | jeblair: ok ^^ that's cleaned up a little from the garbage I had at first - but still is likely open for many improvements | 15:51 |
| *** rcernin has quit IRC | 15:52 | |
| jeblair | brb | 15:55 |
| *** Guest14069 has quit IRC | 15:59 | |
| fungi | wow, we actually get a significant simplification in the bargain | 16:00 |
| openstackgerrit | Merged openstack-infra/project-config: Add new project called molteniron https://review.openstack.org/388186 | 16:00 |
| jeblair | re | 16:00 |
| mordred | it's got some issues ... | 16:01 |
| mordred | anybody know why this: bash -c sudo ls doesn't work? | 16:01 |
| *** david-lyle_ is now known as david-lyle | 16:01 | |
| fungi | quoting | 16:02 |
| fungi | bash -c 'sudo ls' | 16:02 |
| mordred | gotcha | 16:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental ironic grenade multitenant job https://review.openstack.org/388239 | 16:03 |
| *** srobert has joined #openstack-infra | 16:03 | |
| dhellmann | mordred : the handling of args as a list and a string in that function is confusing. It seems to convert back and forth a couple of times. | 16:04 |
| *** jpich has quit IRC | 16:04 | |
| mordred | oh yeah - this code is currently _bad_ | 16:04 |
| *** tqtran has joined #openstack-infra | 16:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Freezer: Fixed tempest regex https://review.openstack.org/388076 | 16:06 |
| mordred | jeblair, fungi: ok. it worked for a little bit, but now I'm back to it not working - trying to figure out what I broke | 16:07 |
| * dhellmann is envisioning a CommandLine class to hide all of this back-and-forth and manipulation | 16:07 | |
| *** amotoki has joined #openstack-infra | 16:07 | |
| *** piet_ has quit IRC | 16:08 | |
| fungi | i get the impression some of the structure there is inherited from similar mess in the ansible module from which it's forked | 16:08 |
| *** nicolasbock has joined #openstack-infra | 16:09 | |
| dhellmann | yeah | 16:09 |
| *** chandankumar has quit IRC | 16:09 | |
| *** armax has joined #openstack-infra | 16:10 | |
| fungi | (note the copyright header and license there) | 16:10 |
| fungi | also the leading comment block explains | 16:10 |
| mordred | jeblair: did you just run a copy of something? | 16:11 |
| jeblair | mordred: i'm occasionally running a python script containing little more than 'pty.spawn(['sudo', 'ls'], follow, read)' as the jenkins user on the proposal slave. | 16:12 |
| *** yamahata has joined #openstack-infra | 16:12 | |
| mordred | jeblair: ok. cool | 16:12 |
| mordred | jeblair: I saw a log entry happen in console.log and wasn't sure if it was you or not | 16:13 |
| jeblair | ah, nope. | 16:13 |
| mordred | so - I'm back to sudo prompting for a password - because it now has a tty and will happily prompt for password | 16:14 |
| jeblair | yeah, the most progress i've been able to make with pty.spawn is that i can capture output from the pty. | 16:15 |
| mordred | yah. I swear I actually saw this work, but I'm starting to doubt my own sanity | 16:15 |
| jeblair | * Lookup controlling tty for this process via sysctl. | 16:18 |
| jeblair | * This will work even if std{in,out,err} are redirected. | 16:18 |
| jeblair | from sudo ^ | 16:18 |
| *** openstackgerrit has quit IRC | 16:18 | |
| *** openstackgerrit has joined #openstack-infra | 16:19 | |
| *** maeker has joined #openstack-infra | 16:19 | |
| fungi | indeed, in ttyname.c | 16:19 |
| *** oanson has quit IRC | 16:19 | |
| jeblair | that at least explains why it's able to get a pty regardless of stdin/out. i don't understand enough about python pty module yet to know if its fork tricks are enough to get around that | 16:20 |
| *** cody-somerville has quit IRC | 16:21 | |
| *** cody-somerville has joined #openstack-infra | 16:21 | |
| *** cody-somerville has joined #openstack-infra | 16:21 | |
| *** simondodsley has joined #openstack-infra | 16:22 | |
| jeblair | https://github.com/ansible/ansible/issues/14377 is relevant | 16:23 |
| jeblair | (ansible runs with 'ssh -tt') | 16:23 |
| *** derekh has quit IRC | 16:25 | |
| *** vsaienko has joined #openstack-infra | 16:27 | |
| jeblair | so... if we *are* running with pipelining, it *doesn't* use -tt ? | 16:27 |
| *** nherciu has joined #openstack-infra | 16:27 | |
| mordred | jeblair: I see -tt either way | 16:28 |
| *** matrohon has quit IRC | 16:28 | |
| persia | Could this be worked around by using ssh-agent? | 16:28 |
| *** mriedem has quit IRC | 16:30 | |
| *** mriedem has joined #openstack-infra | 16:30 | |
| jeblair | mordred: if i run a simple popen test, i get the behavior we want | 16:31 |
| jeblair | mordred: if i remove -tt | 16:31 |
| mordred | jeblair: oh lovely | 16:32 |
| *** baoli has joined #openstack-infra | 16:33 | |
| *** pilgrimstack has quit IRC | 16:34 | |
| mordred | jeblair: so - looking at the ansible source | 16:34 |
| mordred | the function that runs this passes -tt: if not in_data and sudoable: | 16:34 |
| jeblair | mordred: so if in_data is set, we get -tt. but in_data is set if we pipeline? that seems backwards. | 16:36 |
| *** yolanda has quit IRC | 16:36 | |
| jeblair | oh, no i said that backwards. | 16:36 |
| mordred | yah | 16:37 |
| jeblair | if in_data is set we do not get -tt. which does make sense. | 16:37 |
| mordred | yah | 16:37 |
| *** jtomasek has quit IRC | 16:38 | |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| *** amitgandhinz has joined #openstack-infra | 16:39 | |
| mordred | jeblair: I am not finding a great place to hook/override that behavior - other than making either an action plugin that does evil, or an ssh connection plugin that does evil | 16:41 |
| *** makowals has joined #openstack-infra | 16:41 | |
| jeblair | mordred: do you grok why setting pipelining=true doesn't trigger it? | 16:41 |
| *** mkoderer has quit IRC | 16:42 | |
| *** Apoorva has joined #openstack-infra | 16:42 | |
| mordred | jeblair: no, I do not | 16:42 |
| *** tpsilva has quit IRC | 16:44 | |
| AJaeger | clarkb: did you figure out the solum problem? Do we need to restart gerrit? | 16:49 |
| clarkb | AJaeger: I think it is beginning to look that way | 16:50 |
| jeblair | mordred: whether i set pipelining=true, i still see 4 ssh operations for a simple command (mkdir, put, chmod, exec). i would expect that to be one with pipelining enabled, right? | 16:52 |
| *** makowals has quit IRC | 16:53 | |
| jlk | hrm. | 16:53 |
| jlk | it should be yes | 16:53 |
| jlk | unless it's a special module | 16:53 |
| mordred | jeblair: is it actually 4 different commands? or is it just logging each of the actions it's taking separately | 16:53 |
| *** BobBall is now known as BobBall_AWOL | 16:53 | |
| jlk | but yes, pipeline=true should not do the separated actions | 16:54 |
| jeblair | mordred: well, it shouldn't be doing the first the commands i think | 16:54 |
| jeblair | er 'first three commands' | 16:54 |
| mordred | oh right. good point | 16:54 |
| jlk | jeblair: where are you setting pipelining=true ? | 16:54 |
| *** amitgandhinz has quit IRC | 16:55 | |
| jeblair | [ssh_connection] section of ansible.cfg in CWD | 16:55 |
| jlk | alright | 16:55 |
| *** amitgandhinz has joined #openstack-infra | 16:55 | |
| jlk | just ruling that out. | 16:56 |
| mordred | jlk: C.DEFAULT_KEEP_REMOTE_FILES | 16:56 |
| jlk | and connection method/plugin is ssh ? | 16:56 |
| mordred | gah | 16:56 |
| mordred | jeblair: C.DEFAULT_KEEP_REMOTE_FILES has an effect on this | 16:56 |
| jlk | It does, if you are setting that | 16:56 |
| jlk | but I found that if you have pipelining that setting KEEP_REMOTE_FIELS doesn't work | 16:56 |
| *** zz_dimtruck is now known as dimtruck | 16:56 | |
| jlk | at least I thought so | 16:56 |
| jlk | Keeping remote files defaults to off, so unless you're setting it during execution... | 16:57 |
| mordred | I just unset it and now pipelining seems to work | 16:57 |
| jeblair | yeah, disabling keep_remote_files fixes it | 16:57 |
| *** vsaienko has quit IRC | 16:57 | |
| jeblair | apparently that takes precedence over pipelining | 16:57 |
| mordred | good to know | 16:57 |
| jlk | ah interesting | 16:57 |
| jeblair | (we had it set because we started with our current config, where we have that set to avoid an async bug) | 16:57 |
| mordred | I guess it can't keep remote files if it never made any | 16:57 |
| jlk | guess it's been a while since I played with both | 16:57 |
| fungi | that's a surprising side effect | 16:57 |
| jlk | well | 16:58 |
| jlk | with pipelining there are no remote files | 16:58 |
| jlk | so it almost makes sense | 16:58 |
| mordred | jlk: yah. exactly | 16:58 |
| fungi | right, so it's effectively disabling pipelining | 16:58 |
| jeblair | i mean "error! does not compute! printer on fire!" might be a better behavior | 16:58 |
| mordred | jeblair: so - with that removed, I no-longer see -tt in the args | 16:58 |
| jlk | I still think that should be a warning or error on conflicting configs | 16:58 |
| mordred | jlk: ++ | 16:58 |
| fungi | agreed, conflicting options shouldn't result in undefined behavior | 16:58 |
| *** e0ne has quit IRC | 16:58 | |
| mordred | jeblair: WOOT! | 16:59 |
| jeblair | 2016-10-20 16:59:48.408057 | sudo: no tty present and no askpass program specified | 16:59 |
| jeblair | 2016-10-20 16:59:48.408755 | [Zuul] Task exit code: 1 | 16:59 |
| mordred | jeblair: turning off remote_files and putting our current command module back in place looks like it works | 17:00 |
| fungi | that's a relief | 17:00 |
| jeblair | mordred: agreed | 17:00 |
| mordred | wow. | 17:00 |
| mordred | that was very hard - but thankfully is a very easy fix | 17:00 |
| *** davidsha has joined #openstack-infra | 17:00 | |
| jeblair | yeah, and it all boiled down to "we failed to turn pipelining on" | 17:00 |
| fungi | so basically we can disable an option that's no longer necessary anyway and things do what we wanted? best possible outcome | 17:00 |
| mordred | jeblair: you wanna do the honors or shall I? | 17:00 |
| davidsha | would this be the correct place to ask questions about making a new project? | 17:01 |
| fungi | davidsha: sure, assuming it starts with you saying you've read http://docs.openstack.org/infra/manual/creators.html | 17:01 |
| jeblair | mordred, jlk: i think there's two things to take back to the ansible community: 1) something about these two conflicting options. 2) we really want to keep the ability to run without a tty. some folks have attempted to "fix" the pipelining module so it always runs with a tty, but we're going to want to be able to intentionally run without one. | 17:02 |
| mordred | jeblair: ++ | 17:02 |
| jeblair | mordred: yeah, i'll push up a fix | 17:02 |
| davidsha | fungi: Yes, I've just finished making the launchpad and I'm about to move onto pypi | 17:02 |
| fungi | awesome. what's the question? | 17:03 |
| *** yamamoto has quit IRC | 17:03 | |
| *** yamamoto has joined #openstack-infra | 17:03 | |
| *** yamamoto has quit IRC | 17:03 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add external_ipv4_floating_networks https://review.openstack.org/386551 | 17:03 |
| *** mhickey has quit IRC | 17:03 | |
| davidsha | The project I'm making isn't for something to publish, it's an example project for how to make an out of tree neutron extension. I was just wondering would I skip the pypi step? | 17:03 |
| *** ralonsoh has quit IRC | 17:03 | |
| *** ijw has joined #openstack-infra | 17:04 | |
| fungi | davidsha: yeah, if you're not going to have it build a package that's uploaded to and installable from pypi, then there's no need | 17:05 |
| fungi | davidsha: though that said, you ought to consider whether it's possible to make that a cookiecutter template | 17:05 |
| anteaya | there seem to be a number of cookiecutters on pypi: https://pypi.python.org/pypi/cookiecutter/1.4.0 | 17:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: remove keep_remote_files https://review.openstack.org/389280 | 17:06 |
| fungi | davidsha: we already have several cookiecutter template repos for things similar to what you may be doing (for example, general python projects, oslo libraries, puppet modules, et cetera) | 17:06 |
| jeblair | mordred, fungi, jlk, pabelanger: ^ i'm going to test 389280 real quick | 17:06 |
| fungi | sounds good | 17:06 |
| fungi | thanks jeblair | 17:06 |
| mordred | jeblair: I fully support that patch | 17:07 |
| mordred | jeblair: and I am VERY glad that the answer was not "write an ssh connection plugin" | 17:07 |
| jlk | Been down that path, don't want to go back | 17:07 |
| jlk | actually, still on that path :( | 17:07 |
| davidsha | fungi: True, that sounds like a good Idea. Though I'm giving a presentation with this and would like to have it hosted somewhere for people to download it and test it. Since it's extending Neutron and could be updated as new features are added for extensions I thought it might be better as a project. | 17:08 |
| * jlk mutters about locking around host keys | 17:08 | |
| *** ihrachys has quit IRC | 17:08 | |
| anteaya | davidsha: the project/repo could be a cookiecutter template | 17:08 |
| anteaya | is I believe what the suggestion is that is being offered | 17:09 |
| fungi | though we do also have precedent for similar "example" repos | 17:09 |
| fungi | e.g., http://git.openstack.org/cgit/openstack-infra/project-config-example/ | 17:09 |
| mordred | jlk: ugh hostkeyes | 17:09 |
| anteaya | for example: http://git.openstack.org/cgit/openstack/tempest-plugin-cookiecutter/ | 17:09 |
| mordred | jlk: have I mentioned "why can't my cloud provide me a hostkey" recently? | 17:09 |
| *** inc0 has quit IRC | 17:10 | |
| fungi | now that we have "just give me a network" we need "just give me a host key" ;) | 17:10 |
| jeblair | mordred, fungi, jlk: that patch looks good in my local testing so i have approved it | 17:10 |
| *** jpena is now known as jpena|off | 17:10 | |
| mordred | jeblair: ++ | 17:10 |
| *** inc0 has joined #openstack-infra | 17:11 | |
| *** martinkopec has quit IRC | 17:11 | |
| mordred | jeblair: it jives with the results of my testing as well | 17:11 |
| fungi | jeblair: thanks, it was a refreshingly simple patch after half a day of confusion | 17:11 |
| * mordred goes to abandon the pty.spawn patch | 17:11 | |
| davidsha | anteaya, fungi: would a cookie cutter template have the same bug filing systems as other openstack projects? | 17:11 |
| fungi | davidsha: sure, it's a repo just like any other | 17:12 |
| anteaya | davidsha: I don't see why it wouldn't | 17:12 |
| jeblair | patches that remove code and add comments are my favorites | 17:12 |
| mordred | jeblair: ++ | 17:12 |
| *** trown is now known as trown|lunch | 17:12 | |
| fungi | i guess we can do that global launcher restart once it's everywhere | 17:13 |
| fungi | and after that i'll get together a list of what needs to be rerun | 17:13 |
| davidsha | fungi, anteaya : cool, Is it ok if I come back some time tomorrow after looking up what I'd need to change to make it a cookie cutter template? | 17:13 |
| fungi | davidsha: there are people around in here all the time, so sure | 17:13 |
| davidsha | fungi, anteaya: Cool, thanks for the input! | 17:14 |
| anteaya | davidsha: thanks for asking | 17:14 |
| fungi | though we will likely start getting scarce as we move into the weekend and some people start getting on flights | 17:14 |
| mordred | fungi: I thought you said "start getting sarcasm" - and I was like "when did we ever stop?" | 17:15 |
| fungi | at least now you have an appropriately sarcastic comeback already formulated, should i ever happen to say it | 17:16 |
| Zara | :D | 17:16 |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: remove keep_remote_files https://review.openstack.org/389280 | 17:16 |
| dhellmann | fungi, mordred : what's the shelf-life of canned sarcasm? | 17:17 |
| mordred | dhellmann: I think if properly bottled it can last indefinitely | 17:17 |
| dhellmann | jeblair, mordred, fungi : am I correct in reading the scrollback to mean that the config option change fixes the sudo issue in the release jobs? | 17:17 |
| dhellmann | mordred : or pickled, I suppose | 17:18 |
| mordred | dhellmann: yes. | 17:18 |
| fungi | dhellmann: or at least it will once it's in place and restarts happen | 17:18 |
| dhellmann | ok, that was my next question :-) | 17:18 |
| *** flepied has quit IRC | 17:19 | |
| fungi | dhellmann: in parallel though, do you have a list of what failed so i can get ready to rerun stuff that needs it once the restart is behind us? | 17:19 |
| dhellmann | fungi : how long does the restart take? is it just zuul? | 17:19 |
| dhellmann | fungi : https://etherpad.openstack.org/p/6mZZeAigiR | 17:19 |
| fungi | it'll be a restart of the zuul launchers, though i think we just need the zlstatic01 restarted for me to be able to start in on release jobs | 17:20 |
| dhellmann | let me see if I can add version numbers to the repos that don't have them | 17:20 |
| fungi | jeblair: ^ ? | 17:20 |
| jeblair | fungi: true, but it'll probably be easier to hard restart all at once | 17:20 |
| clarkb | fungi: assuming that all of the jobs you need rerun run on static hosts yes. But tarball builds etc run on general instances | 17:20 |
| fungi | cool, didn't know if you were going to shoot for a graceful rolling restart instead | 17:20 |
| jeblair | should be done in < 30 mins | 17:21 |
| fungi | clarkb: in tis case the only observed issue for release work was that jobs running on static nodes with the revoke-sudo builder were hanging and timing out | 17:21 |
| jeblair | fungi, clarkb, mordred: i will manually update zuul, install, and restart on the launchers | 17:21 |
| jeblair | rather than waiting for puppet to oom. | 17:21 |
| fungi | because the jenkins user doesn't actually have permission to revoke sudo for itself on those nodes, as we don't ever grant it | 17:21 |
| fungi | jeblair: thanks, that'll speed things along nicely | 17:22 |
| *** e0ne has joined #openstack-infra | 17:22 | |
| clarkb | sounds good (also its ansiblet hat ooms not puppet aiui) | 17:23 |
| *** degorenko is now known as _degorenko|afk | 17:23 | |
| *** jcoufal has joined #openstack-infra | 17:23 | |
| *** sputnik13 has joined #openstack-infra | 17:24 | |
| *** baoli_ has joined #openstack-infra | 17:24 | |
| *** jcoufal_ has quit IRC | 17:24 | |
| mordred | pansible ooms | 17:25 |
| *** baoli has quit IRC | 17:25 | |
| mordred | that almost sounds like a band | 17:25 |
| AJaeger | team, clarkb has been trying t ofigure out the many "Submitted, Merge Pending" in solum - and we think it's best to restart gerrit, seems there's some jgit corruption. When do you want to restart gerrit? | 17:25 |
| hamzy | Hello, could someone please add me to molteniron-core and molteniron-release? Thanks! | 17:26 |
| jeblair | #status log restarted ansible launchers with 2.5.2.dev31 | 17:26 |
| openstackstatus | jeblair: finished logging | 17:26 |
| jeblair | fungi, dhellmann: should be gtg | 17:26 |
| fungi | thanks jeblair! | 17:27 |
| dhellmann | thanks! | 17:28 |
| *** davidsha has quit IRC | 17:28 | |
| dhellmann | fungi : is the info in that etherpad clear and complete for restarting those jobs? I added version numbers to the lines that were missing them. | 17:28 |
| dhellmann | I think it's safe to ignore the wheel errors | 17:28 |
| *** yaume has quit IRC | 17:29 | |
| fungi | dhellmann: yeah, i'll need to look up the shas of those tags, but it looks like i can just reenqueue the tag refs for all of them | 17:29 |
| dhellmann | I can start pulling shas for you | 17:30 |
| fungi | or i can nab them from the log urls you're adding | 17:30 |
| dhellmann | ok, the log urls are easy | 17:30 |
| fungi | perfect | 17:30 |
| *** esikache1 has joined #openstack-infra | 17:31 | |
| *** esikache1 has quit IRC | 17:31 | |
| *** esikachev has joined #openstack-infra | 17:32 | |
| *** tkelsey has quit IRC | 17:33 | |
| *** mat128 is now known as mat128|afk | 17:33 | |
| *** jcoufal has quit IRC | 17:34 | |
| *** tiswanso has joined #openstack-infra | 17:35 | |
| AJaeger | OOps, we have still 70 periodic jobs running - including the requirements one from yesterday ;( | 17:38 |
| AJaeger | harlowja: http://logs.openstack.org/periodic/periodic-keystone-py27-with-oslo-master/de09219/console.html is failing ;( | 17:39 |
| dhellmann | fungi : for js-openstack-lib, are you the owner then or should I talk to someone else? | 17:40 |
| *** sputnik13 has quit IRC | 17:40 | |
| fungi | dhellmann: okay, hopefully they're all reenqueued correctly now, except for js-openstack-lib which needs a 0.0.2 tag pushed (for some reason npm thought 0.0.1 was already released/uploaded and refused the upload) | 17:40 |
| AJaeger | fungi, before you enqueue, please check status of zuul - I'm surprised by those 70 periodic jobs... | 17:40 |
| dhellmann | fungi : it would be even better to go with 0.1.0, but sure | 17:40 |
| fungi | AJaeger: i'm betting those are due to lengthy timeouts for jobs with revoke-sudo on static nodes but looking now | 17:41 |
| *** sputnik13 has joined #openstack-infra | 17:41 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: zypper: Do not pull recommended packages https://review.openstack.org/388847 | 17:41 |
| fungi | dhellmann: yeah, i honestly don't know what the versioning conventions are for the nodejs/npm ecosystem, so am mostly relying on cardeois to tell me (and the packaging.json file in it needs a commit updating it to the next release version before we tag it) | 17:42 |
| cardeois | fungi dhellmann I just merged the 0.0.2 version | 17:43 |
| dhellmann | fungi : ok, if they have their own conventions we should follow those. I discourage 0.0.x versions in our projects because it makes branching slightly more confusing and implies the first release has no features. I could be over-pedantic on that, though. | 17:43 |
| cardeois | can you maybe try to replublish please? | 17:43 |
| AJaeger | fungi: ah, that might be... | 17:43 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Hide arrows to expand task details if there are no details https://review.openstack.org/379417 | 17:43 |
| cardeois | fungi: https://review.openstack.org/#/c/388940/ | 17:44 |
| fungi | dhellmann: looks like the reenqueued releases are succeeding rather than timing out now, so we're hopefully all set there | 17:44 |
| *** SumitNaiksatam has joined #openstack-infra | 17:44 | |
| fungi | jeblair: mordred: jlk: ^ !!! | 17:44 |
| dhellmann | fungi : yep, I'll keep an eye on these and if they all run through then I'll approve some of the other outstanding items we put on hold | 17:44 |
| dhellmann | fungi, jeblair, mordred, jlk, pabelanger : thank you all for getting to the bottom of this issue today and fixing it! | 17:45 |
| harlowja | AJaeger thx, will have to see if i can figure out why that's dying | 17:45 |
| mordred | fungi: yay!!! | 17:45 |
| *** rtheis has joined #openstack-infra | 17:46 | |
| *** vsaienko has joined #openstack-infra | 17:48 | |
| AJaeger | fungi, feel free to kill the periodic jobs if needed - before they run several days | 17:48 |
| fungi | AJaeger: yeah, looks like we're starved on translation update jobs, which all run on one static node so we probably timed out a bunch of them and periodic has a lower priority | 17:48 |
| fungi | i expect it will catch back up quickly now that the fix is in place | 17:48 |
| AJaeger | fungi, it normally takes 3-4 hours to run them - so, let's see. Also, there are entries in release queue from yesterday. Hope they move forward as well... | 17:49 |
| AJaeger | harlowja: there are more failing like http://logs.openstack.org/periodic/periodic-glance-py27-with-oslo-master/74402b2/ | 17:50 |
| AJaeger | harlowja: but some of these might also be due to the problems that the team just fixed (not the keystone one) | 17:51 |
| harlowja | i also recently changed that jenkins script | 17:51 |
| AJaeger | harlowja: I know - that's why I point it out ;) | 17:51 |
| *** dimtruck is now known as zz_dimtruck | 17:51 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 17:52 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Add StoryBoard integration for graph commands https://review.openstack.org/385112 | 17:52 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Add Jira integration for graph commands https://review.openstack.org/385217 | 17:52 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Wait until the most recent index is available https://review.openstack.org/387986 | 17:52 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Refactor launchpad code into bug_tracker https://review.openstack.org/385199 | 17:52 |
| harlowja | AJaeger :) | 17:52 |
| harlowja | something in https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-tox-with-oslo-master.sh#L78-L99 i guess | 17:52 |
| AJaeger | fungi, clarkb has been trying to figure out the many "Submitted, Merge Pending" in solum - and we think it's best to restart gerrit, seems there's some jgit corruption. When should we restart gerrit? And who can do it? | 17:52 |
| harlowja | seems to be getting past the 'diff' part, so that's good | 17:52 |
| AJaeger | harlowja: hope you figure it out, it's only 22 lines ;) | 17:52 |
| harlowja | :-p | 17:52 |
| *** onovy has quit IRC | 17:52 | |
| *** peterlisak has quit IRC | 17:52 | |
| *** zz_dimtruck is now known as dimtruck | 17:53 | |
| harlowja | AJaeger ya, i think i got it | 17:53 |
| harlowja | the diff command seems to exit with non-zero if there is a diff | 17:53 |
| harlowja | didn't expect that... | 17:54 |
| *** tphummel has joined #openstack-infra | 17:54 | |
| AJaeger | we use bash -xe normally, don't we? | 17:54 |
| harlowja | ya, tox does also i think | 17:55 |
| clarkb | AJaeger: fungi everythign I can tell is that the git repo itself is fine so must be some issue with jgit's current state? Similar to what we saw with eg nova and that failed upgrade | 17:56 |
| *** markvoelker has quit IRC | 17:56 | |
| fungi | we can probably restart gerrit any time. i'm mildly surprised that we haven't needed to restart recently for jvm gc reasons | 17:56 |
| *** onovy has joined #openstack-infra | 17:57 | |
| *** peterlisak has joined #openstack-infra | 17:58 | |
| fungi | cardeois: looks good! http://logs.openstack.org/3b/3b90b54528bca46aac55a89eaa29cc2df18805e7/release/js-openstack-lib-npm-upload/b85c45a/console.html | 17:58 |
| fungi | cardeois: and https://www.npmjs.com/package/openstack-lib shows 0.0.2 as the latest release now | 17:58 |
| cardeois | awesome thanks a lot fungi ! | 17:58 |
| fungi | happy to help | 17:59 |
| AJaeger | fungi, clarkb , could either of you do it,please? | 18:00 |
| *** amoralej is now known as amoralej|off | 18:00 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Ensure a diff does not cause a non-zero return code https://review.openstack.org/389301 | 18:01 |
| harlowja | AJaeger ok ^ should help with the diff problem, lol | 18:01 |
| *** amitgandhinz has quit IRC | 18:01 | |
| *** amitgandhinz has joined #openstack-infra | 18:02 | |
| fungi | AJaeger: probably want to make sure dhellmann is done or paused approving more release changes first | 18:03 |
| fungi | also i need to step away for lunch/voting so will be gone for a few hours | 18:03 |
| fungi | better if i'm not the one to do the restart | 18:03 |
| dhellmann | AJaeger, fungi : I'm about to tag the final releases for the cycle-trailing projects. Should I wait? | 18:03 |
| fungi | i suppose i can do the gerrit restart right now before i head out | 18:04 |
| *** yamamoto has joined #openstack-infra | 18:04 | |
| fungi | clarkb: are you around to look into this after i restart gerrit? | 18:04 |
| fungi | infra-root: any objections to a gerrit restart so we can hopefully clear the submitted-merge-pending solum changes? | 18:04 |
| fungi | looks like our most recent gerrit restart was 10 days ago | 18:05 |
| dhellmann | fungi : I'll stand by and wait for that restart | 18:05 |
| AJaeger | thanks, fungi! | 18:06 |
| clarkb | I am fairly distracted too | 18:06 |
| fungi | i'll #status notice it and fire the restart now | 18:06 |
| clarkb | trying to get pretrip stuff done today so that I have time for work tomorrow... | 18:06 |
| *** tqtran has quit IRC | 18:06 | |
| fungi | and then i'll make sure it comes back up and is working at least | 18:07 |
| fungi | #status notice The Gerrit service on review.openstack.org is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily. | 18:07 |
| openstackstatus | fungi: sending notice | 18:07 |
| *** mountpoint has joined #openstack-infra | 18:07 | |
| *** flepied has joined #openstack-infra | 18:07 | |
| *** maishsk has joined #openstack-infra | 18:08 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily. | 18:08 | |
| fungi | webui is returning content again | 18:08 |
| *** tphummel has quit IRC | 18:08 | |
| fungi | looks like it's back to working order | 18:08 |
| fungi | AJaeger: have one of those broken solum changes handy? | 18:08 |
| AJaeger | does the solum team need to do anything to get the changes merged? recheck? | 18:08 |
| AJaeger | https://review.openstack.org/#/q/project:openstack/solum | 18:09 |
| AJaeger | pick one ;) | 18:09 |
| AJaeger | https://review.openstack.org/384199 | 18:09 |
| AJaeger | the first fungi^ | 18:09 |
| openstackstatus | fungi: finished sending notice | 18:10 |
| fungi | yeah, looks like they're still showing as in a submitted, merge pending state | 18:10 |
| jlk | mordred: yeah, hostkey would be good to get from cloud. But that doesn't help Ansible. | 18:10 |
| fungi | AJaeger: not sure what the next step is | 18:10 |
| *** e0ne has quit IRC | 18:10 | |
| AJaeger | I could recheck one... | 18:10 |
| bswartz | got a question for you infra guys -- I'm going to propose addition of a new repo which contains code for testing manila, and it's GPL licensed -- the questions is whether it belongs in the openstack or openstack-infra namespace -- it's not clear from reading http://governance.openstack.org/reference/licensing.html | 18:10 |
| fungi | AJaeger: have you looked to see if any of these changes are actually merged in the repo and just not reflected as merged in gerrit? we can probably fix that in the db or something if so | 18:11 |
| * AJaeger checks | 18:11 | |
| *** yamamoto has quit IRC | 18:11 | |
| *** electrofelix has quit IRC | 18:11 | |
| AJaeger | head is from Oct 9 - so, not merged | 18:12 |
| fungi | bswartz: good question, i think the purpose of the software is what matters, and not necessarily which project team controls it | 18:12 |
| *** timello has quit IRC | 18:12 | |
| bswartz | yeah but that doesn't answer my namespace question | 18:12 |
| bswartz | is the openstack-infra namespace only for stuff your team owns? | 18:12 |
| bswartz | or is it for stuff that's not released as part of openstack? | 18:13 |
| fungi | oh, got it... well, we have infra team projects in the openstack namespace, and there's at least one non-infra project in the openstack-infra namespace | 18:13 |
| AJaeger | bswartz: openstack/ namespace is for everybody, just use it | 18:13 |
| fungi | honestly it's not entirely consistent | 18:13 |
| bswartz | AJaeger: that's what I was leaning towards | 18:13 |
| fungi | but i agree with AJaeger, openstack namespace makes sense | 18:13 |
| *** trown|lunch is now known as trown | 18:13 | |
| bswartz | okay thx | 18:14 |
| jeblair | i think the others should be retired, just like stackforge. it's just a lot of work. | 18:14 |
| fungi | if it turns out that it _needs_ to be under the infra team for some reason, then that's not a reason to change namespaces on it anyway | 18:14 |
| *** otherwiseguy has quit IRC | 18:14 | |
| bswartz | jeblair: including openstack-infra and openstack-dev? | 18:14 |
| fungi | bswartz: yes | 18:14 |
| fungi | ideally we'd just have no namespaces at all, but for mirroring to github we need one | 18:14 |
| *** otherwiseguy has joined #openstack-infra | 18:14 | |
| bswartz | I for one welcome our own single-namespace overlords | 18:14 |
| jeblair | fungi: i think gerrit handles pending merges as soon as it starts up, but just in case, i'd probably wait until it flushes the queue to finally determine whether the restart was effective or not | 18:14 |
| AJaeger | fungi, I rechecked the oldest solum change now | 18:15 |
| bswartz | I for one welcome our new* single-namespace overlords | 18:15 |
| fungi | jeblair: yeah, it may still update it | 18:15 |
| fungi | okay, i'm going to go run my lunch errands now. back in a while | 18:15 |
| AJaeger | jeblair: ah, soI was impatient - let's see... | 18:15 |
| AJaeger | fungi, thanks! | 18:15 |
| dhellmann | fungi, AJaeger : so it's safe to approve patches in gerrit? | 18:15 |
| AJaeger | dhellmann: yes | 18:15 |
| *** sputnik13 has quit IRC | 18:15 | |
| dhellmann | AJaeger : thanks | 18:15 |
| njohnston | Hi! I was hoping the gerrit restart would fix this, but it hasn't. I get "Code Review - Error 500 Internal server error" when I go to https://review.openstack.org/#/c/387398/ and try to cherry-pick it to stable/newton. Please, could someone take a look at the gerrit logs and see what is up? | 18:16 |
| *** kjackal_ has quit IRC | 18:16 | |
| *** kjackal_ has joined #openstack-infra | 18:18 | |
| *** dave-mccowan has joined #openstack-infra | 18:18 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: zypper: Do not pull recommended packages https://review.openstack.org/388847 | 18:24 |
| AJaeger | clarkb, fungi: https://review.openstack.org/#/c/384200/ shows still "Conflicts With (N/A) - 500 Internal Server Error". Let's see whether anything merges eventually... | 18:25 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/project-config: Remove stress jobs from the gate https://review.openstack.org/389308 | 18:26 |
| *** timello has joined #openstack-infra | 18:27 | |
| *** rossella_s has quit IRC | 18:28 | |
| *** rossella_s has joined #openstack-infra | 18:29 | |
| dhellmann | AJaeger, jeblair, fungi : I'm seeing a new release tagging failure :-( http://logs.openstack.org/b3/b3f4eb6d17737c58ec9374cf46d306ad18e7d867/release/instack-undercloud-tarball/77ac87d/console.html | 18:29 |
| AJaeger | ;/ | 18:29 |
| dhellmann | maybe that's a git cache issue? | 18:30 |
| dhellmann | the tag is there | 18:30 |
| AJaeger | dhellmann: strange, that code has not been changed for some time | 18:31 |
| dhellmann | yeah, I suspect an issue with getting the local copy of the git repo updated | 18:31 |
| dhellmann | the repo for the project the job is running for, openstack/instack-undercloud | 18:32 |
| dhellmann | I'm getting that same error with *lots* of repos | 18:32 |
| AJaeger | dhellmann: a git update? | 18:33 |
| AJaeger | git branch -a --contains refs/tags/5.0.0 works for me on thta repo | 18:33 |
| *** vsaienko has quit IRC | 18:33 | |
| dhellmann | AJaeger : yes, same here | 18:33 |
| AJaeger | but http://logs.openstack.org/b3/b3f4eb6d17737c58ec9374cf46d306ad18e7d867/release/instack-undercloud-tarball/77ac87d/console.html#_2016-10-20_18_23_17_628628 says "error: malformed object name refs/tags/5.0.0" | 18:33 |
| dhellmann | that error message means that the tag isn't present locally | 18:33 |
| dhellmann | so the node running that job did not fetch the tags correctly | 18:34 |
| AJaeger | dhellmann: Ah, indeed - can get the error with another tag | 18:34 |
| dhellmann | at one point in the past jeblair added some extra logic to zuul-cloner to ensure that all tags were fetched | 18:34 |
| AJaeger | jeblair: zuul-cloner did not fetch the 5.0.0 tag ^ | 18:34 |
| dhellmann | at least I think so? I know we had a work-around in some other release jobs | 18:34 |
| AJaeger | dhellmann: I have no idea and can't help you further, hope others can. | 18:35 |
| dhellmann | AJaeger : thanks | 18:35 |
| *** tnovacik has quit IRC | 18:35 | |
| AJaeger | infra-root, could you help dhellmann, please? ^ | 18:35 |
| *** timello has quit IRC | 18:36 | |
| *** rtheis has quit IRC | 18:36 | |
| *** ociuhandu has quit IRC | 18:38 | |
| *** pilgrimstack has joined #openstack-infra | 18:39 | |
| *** mountpoint has quit IRC | 18:40 | |
| dhellmann | interesting, I see that some jobs did ok but some failed and of those it looks like most (all?) are on osic nodes | 18:41 |
| dhellmann | I have no idea if that's meaningful | 18:41 |
| *** dimtruck is now known as zz_dimtruck | 18:42 | |
| irtermite | dhellmann: does it say how/why failed? | 18:42 |
| irtermite | cloudnull: ^^ | 18:43 |
| dhellmann | irtermite : I'm building a list of links to the logs for failures at the bottom of https://etherpad.openstack.org/p/6mZZeAigiR | 18:43 |
| dhellmann | so far they all seem to be failing because they're not seeing the new tag | 18:43 |
| dhellmann | a lot of the failures are doc jobs, which aren't critical | 18:43 |
| dhellmann | but there are 2 tarball jobs that we'll need to redo | 18:44 |
| irtermite | not seeing the 5.0.0 tag specifically, or having trouble hitting the repo period? | 18:44 |
| *** timello has joined #openstack-infra | 18:44 | |
| dhellmann | irtermite : there is no error about trying to fetch from upstream, but after the fetch the tag is not present | 18:44 |
| irtermite | odd | 18:45 |
| dhellmann | irtermite : this is a representative example http://logs.openstack.org/b3/b3f4eb6d17737c58ec9374cf46d306ad18e7d867/release/instack-undercloud-tarball/77ac87d/console.html | 18:45 |
| irtermite | dhellmann: "2016-10-20 18:28:09.764006 | error: malformed object name refs/tags/14.0.0 " | 18:45 |
| irtermite | ??? | 18:46 |
| irtermite | oops, wrong one... "2016-10-20 18:23:17.628628 | error: malformed object name refs/tags/5.0.0 " | 18:46 |
| dhellmann | right. that error means the local copy of the repo does not contain the tag that was just pushed to the upstream copy of the repo by another job, which then triggered the job that failed | 18:46 |
| *** markvoelker has joined #openstack-infra | 18:46 | |
| *** vsaienko has joined #openstack-infra | 18:46 | |
| dhellmann | IOW, the job that failed was triggered by a tag being pushed, but the job itself doesn't see that tag for some reason | 18:47 |
| *** mountpoint has joined #openstack-infra | 18:47 | |
| *** mountpoint has quit IRC | 18:47 | |
| *** markvoelker has quit IRC | 18:47 | |
| *** markvoelker has joined #openstack-infra | 18:47 | |
| dhellmann | this job failed on internap, so I don't think it's related to the cloud where the job ran http://logs.openstack.org/ef/efd9ddd965314be2b8bb41d9c36301db13988aa4/release/openstack-ansible-os_aodh-docs-ubuntu-xenial/0392b2c/console.html | 18:48 |
| jeblair | dhellmann: can you describe the process flow in detail re "the tag that was just pushed to the upstream copy of the repo by another job, which then triggered the job that failed" | 18:48 |
| irtermite | yea, doesn't seem like an error that would be cloud related anyway | 18:48 |
| irtermite | sounds like something is not checking out properly | 18:49 |
| *** mountpoint has joined #openstack-infra | 18:49 | |
| *** mountpoint has quit IRC | 18:49 | |
| *** eharney has quit IRC | 18:49 | |
| *** e0ne has joined #openstack-infra | 18:49 | |
| dhellmann | jeblair : when a patch is merged to openstack/releases the tag-releases job runs in the post-release queue. That job runs on the special signing node with privileges to create signed tags and push them to gerrit. That job ran, and pushed a bunch of tags. Those tags in turn triggered various jobs related to releasing whatever was being tagged. Some of those have failed. | 18:50 |
| *** zzelle has joined #openstack-infra | 18:50 | |
| dhellmann | so far most of the failures look like jobs to rebuild documentation (I think those run in the tag queue) or to announce new releases | 18:50 |
| dhellmann | (I think those run in the releases queue) | 18:50 |
| *** Julien-zte has quit IRC | 18:50 | |
| *** tobias_ has joined #openstack-infra | 18:51 | |
| dhellmann | two of the failed jobs were tarball jobs, though | 18:51 |
| dhellmann | unfortunately, this release included all of the ansible repos, and there are a zillion of those | 18:51 |
| *** zz_dimtruck is now known as dimtruck | 18:52 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver https://review.openstack.org/389317 | 18:52 |
| *** mountpoint has joined #openstack-infra | 18:52 | |
| *** mountpoint has quit IRC | 18:52 | |
| *** mriedem has quit IRC | 18:53 | |
| jeblair | i think i understand the problem | 18:53 |
| *** Julien-zte has joined #openstack-infra | 18:53 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver https://review.openstack.org/388918 | 18:53 |
| dhellmann | jeblair : we had a similar issue a while back, and I made some changes to a few of the release scripts to force fetching tags as part of cloning a repo | 18:54 |
| dhellmann | I think those were in validation jobs or something, I don't remember exactly | 18:54 |
| jeblair | dhellmann: yeah, and that's in zuul-cloner now. that's working, believe it or not. :) | 18:54 |
| dhellmann | but I thought that at the same time you also made changes to zuul-cloner to do the same thing | 18:54 |
| dhellmann | k | 18:54 |
| dhellmann | I wasn't sure if that made it in | 18:54 |
| jeblair | this happened because of the gerrit restart | 18:54 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver https://review.openstack.org/388918 | 18:55 |
| dhellmann | jeblair : that's what I was afraid of | 18:55 |
| AJaeger | jeblair: really? dhellmann waited until gerrit was restarted | 18:56 |
| *** mountpoint has joined #openstack-infra | 18:56 | |
| jeblair | the problem is that we tell zuul-cloner to use git.openstack.org as the canonical location to update a repository from. when a tag is pushed to gerrit, it takes some time to be replicated to all the remote repos (git.o.o and github). normally that's pretty fast. but in this case, it went to the back of a queue that is about 15,000 git push operations long. it takes, i want to say, longer than 30 minutes for gerrit to fully sync ... | 18:57 |
| jeblair | ... all the repos. | 18:57 |
| AJaeger | argh ;( | 18:57 |
| AJaeger | sorry, dhellmann | 18:57 |
| dhellmann | ok, so if I'd waited an hour or so everything would be fine? | 18:57 |
| dhellmann | that's good to know for next time | 18:58 |
| jeblair | the best solution to this is to have zuul wait for replication completed events for these. that may be doable in the long run, but it's complicated (someone started on a change to do that, but i don't think the first cut was quite right) | 18:58 |
| *** lucasagomes is now known as lucas-afk | 18:58 | |
| *** jkilpatr has quit IRC | 18:58 | |
| *** tobias_ has quit IRC | 18:59 | |
| jeblair | that will solve even the small race condition we have in the normal case | 18:59 |
| *** mriedem has joined #openstack-infra | 18:59 | |
| jeblair | (usually, it takes longer to get a job running on a node that it takes for gerrit to push the ref out everywhere, but theoretically, it could be faster) | 18:59 |
| *** amitgandhinz has quit IRC | 18:59 | |
| *** kgiusti has quit IRC | 18:59 | |
| AJaeger | jeblair: yeah, we have too many nodes ;( | 18:59 |
| *** amitgandhinz has joined #openstack-infra | 19:00 | |
| *** woodster_ has joined #openstack-infra | 19:00 | |
| *** panda|bbl is now known as panda | 19:00 | |
| jeblair | alternative, we may be able to have the cloner fetch from the zuul merger in this case... i will think about that as i work on the other issue. | 19:00 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver https://review.openstack.org/388921 | 19:01 |
| jeblair | dhellmann: but yeah, in the mean time, i think adding "ask infra-root if the gerrit processing queue is empty if it was just restarted" to the human protocol might be good | 19:01 |
| jeblair | it is now empty. :) | 19:02 |
| *** kgiusti has joined #openstack-infra | 19:02 | |
| jeblair | (and i think we can now say with some confidence, it did not fix the solum issue) | 19:02 |
| dhellmann | jeblair : ok. Can I get you to re-enqueue the 2 tarball jobs that failed? | 19:02 |
| *** kgiusti has left #openstack-infra | 19:02 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver https://review.openstack.org/388921 | 19:02 |
| jeblair | dhellmann: i can try :) | 19:04 |
| jeblair | dhellmann: what were they? | 19:04 |
| dhellmann | jeblair : see lines 31-34 of https://etherpad.openstack.org/p/6mZZeAigiR | 19:04 |
| *** zzelle has quit IRC | 19:04 | |
| dhellmann | let me get version info for you | 19:04 |
| *** abregman|afk is now known as abregman | 19:05 | |
| jeblair | mordred: where are you on the vars.yaml change? i'm having a lot of trouble with this because i can't see them | 19:06 |
| mordred | jeblair: oh! crap. I'm nowhere. let me be somewhere with it real quick | 19:06 |
| AJaeger | jeblair, clarkb, fungi: Indeed, the solum issue is not fixed ;( | 19:07 |
| jeblair | zuul enqueue-ref --trigger=gerrit --pipeline=release --project=openstack/instack-undercloud-tarball --ref=refs/tags/5.0.0 --newrev=b3f4eb6d17737c58ec9374cf46d306ad18e7d867 | 19:07 |
| dhellmann | jeblair : when you're thinking about zuul features, I would be happy to have some sort of interface to restart jobs like this without bugging infra-root about it. | 19:07 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas tempest jobs for legacy and v1 voting https://review.openstack.org/389320 | 19:07 |
| dhellmann | maybe that exists and I just don't have permission, which is also ok | 19:07 |
| AJaeger | infra-root, https://review.openstack.org/385955 was just rechecked and is still in "Submitted, Merge pending" - and there are more of these for solum at https://review.openstack.org/#/q/project:openstack/solum+status:open | 19:08 |
| jeblair | dhellmann: ack | 19:08 |
| *** eharney has joined #openstack-infra | 19:08 | |
| *** Goneri has quit IRC | 19:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-openstackci: Treat yaml files as plain text https://review.openstack.org/389321 | 19:09 |
| mordred | jeblair: ^^ | 19:09 |
| jeblair | zuul enqueue-ref --trigger=gerrit --pipeline=release --project=openstack/kolla-tarball --ref=refs/tags/3.0.0 --newrev=cc3426a73346aa8b7bbdbe78de3849c0e0c8752b | 19:09 |
| jeblair | dhellmann: do those two commands look right? | 19:09 |
| dhellmann | jeblair : let me check the shas | 19:09 |
| *** devkulkarni has joined #openstack-infra | 19:10 | |
| jeblair | mordred: thx | 19:10 |
| mordred | jeblair: sorry for the delay | 19:10 |
| *** maishsk has quit IRC | 19:10 | |
| dhellmann | jeblair : yes, those look correct | 19:11 |
| jeblair | dhellmann: oops, those are job names, not project names | 19:14 |
| *** jkilpatr has joined #openstack-infra | 19:14 | |
| dhellmann | jeblair : oh, sorry, I didn't notice that | 19:14 |
| jeblair | infra-root: i think i've done more than my share of firefighting this week. i am starving and am *far* behind on summit prep. i'm going to go work on those now, and hope that others are now prepped for summit and can handle anything that comes up. i do not expect to be active further on irc this week. | 19:14 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas tempest jobs for legacy and v1 voting https://review.openstack.org/389320 | 19:14 |
| jeblair | dhellmann: those two are done | 19:15 |
| dhellmann | jeblair : thank you | 19:15 |
| mordred | jeblair: ++ | 19:15 |
| dhellmann | jeblair : enjoy your meal, and see you next week! | 19:15 |
| * dhellmann makes a note to avoid having a release deadline the week before a travel event | 19:15 | |
| dhellmann | although for ocata the final release will be *during* the ptg | 19:16 |
| *** Sukhdev has joined #openstack-infra | 19:18 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/389327 | 19:18 |
| njohnston | dhellmann: perfect time for people to pitch in as part of the horizontal team sessions, perhaps? | 19:18 |
| *** maishsk has joined #openstack-infra | 19:18 | |
| dhellmann | njohnston : it would be, if those were on thursday when our release date falls | 19:18 |
| dhellmann | we have time to move that earlier in the week, I guess | 19:18 |
| odyssey4me | thanks dhellmann, mordred, jeblair, AJaeger and many other names for helping get Newton done | 19:21 |
| odyssey4me | it's been a fantastic cycle | 19:22 |
| odyssey4me | and if you don't mind, I'm going to pour myself a rather stuff drink and leave my laptop alone | 19:22 |
| odyssey4me | *stiff | 19:22 |
| odyssey4me | argh | 19:22 |
| dhellmann | odyssey4me : good plan | 19:22 |
| *** amitgandhinz has quit IRC | 19:23 | |
| *** Apoorva has quit IRC | 19:24 | |
| AJaeger | odyssey4me: go for it! glad to hear that Newton is down for you finally ;) | 19:24 |
| *** jheroux has quit IRC | 19:24 | |
| *** amitgandhinz has joined #openstack-infra | 19:25 | |
| anteaya | odyssey4me: enjoy your drink | 19:26 |
| *** ihrachys has joined #openstack-infra | 19:30 | |
| mordred | odyssey4me: yay drinking! (and also getting cycles done) | 19:30 |
| *** stream10 has joined #openstack-infra | 19:30 | |
| *** jkilpatr has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/389327 | 19:32 |
| *** devkulkarni has quit IRC | 19:32 | |
| *** devkulkarni has joined #openstack-infra | 19:33 | |
| *** mhickey has joined #openstack-infra | 19:34 | |
| *** Apoorva has joined #openstack-infra | 19:36 | |
| *** dimtruck is now known as zz_dimtruck | 19:36 | |
| anteaya | jeblair: safe travels | 19:36 |
| *** dizquierdo has joined #openstack-infra | 19:36 | |
| dhellmann | I'm going to follow jeblair & odyssey4me's lead and drop offline to get ready for travel. I'll see you all next week! | 19:36 |
| anteaya | dhellmann: safe travels to you too | 19:36 |
| *** e0ne has quit IRC | 19:43 | |
| *** tnovacik has joined #openstack-infra | 19:43 | |
| *** pilgrimstack has quit IRC | 19:43 | |
| *** yolanda has joined #openstack-infra | 19:43 | |
| ianw | mordred: just checking ... all the launchers are happy now? When i left last night was just zl01 that needed to be unemergencied, but seems like that's all fixed now | 19:44 |
| mordred | ianw: yup! we're in good shape | 19:45 |
| mordred | ianw: found the hanging problem and fixed that too | 19:45 |
| mordred | so it _should_ all be operating normally | 19:45 |
| *** jkilpatr has joined #openstack-infra | 19:46 | |
| ianw | awesome, thanks | 19:48 |
| *** tykeal has joined #openstack-infra | 19:48 | |
| *** ociuhandu has joined #openstack-infra | 19:52 | |
| hamzy | Hello, could someone please add https://launchpad.net/~mark-hamzy to the group https://review.openstack.org/#/admin/groups/1611,members and https://review.openstack.org/#/admin/groups/1612,members please? | 19:55 |
| pleia2 | hamzy: I'll have a look, what's the email address you use for gerrit? | 19:55 |
| hamzy | pleia2, I think my Ubuntu One account is linked to mark.hamzy@gmail.com | 19:56 |
| pleia2 | hamzy: it's what you have in gerrit, not lp/ubuntu one | 19:57 |
| pleia2 | I see hamzy@us.ibm.com | 19:57 |
| pleia2 | ah, the gmail one is in there too | 19:57 |
| hamzy | yeah, I have both... don't know what is primary | 19:57 |
| pleia2 | hamzy: can go here to see: https://review.openstack.org/#/settings/ | 19:57 |
| pleia2 | (while logged in) | 19:57 |
| hamzy | it says hamzy@us.ibm.com | 19:58 |
| pleia2 | ok, adding | 19:58 |
| hamzy | and Account ID18242 | 19:58 |
| hamzy | thanks! | 19:58 |
| pleia2 | hamzy: done :) | 19:58 |
| hamzy | \o/ | 19:59 |
| *** abregman is now known as abregman|afk | 20:01 | |
| vsaienko | devstack-gate core team, please help to merge chain needed for ironic multinode job https://review.openstack.org/#/c/364830 It is a blocker of ironic team. Thanks! | 20:02 |
| *** Naeil has quit IRC | 20:05 | |
| *** Apoorva_ has joined #openstack-infra | 20:07 | |
| *** abregman|afk has quit IRC | 20:08 | |
| *** nherciu has quit IRC | 20:10 | |
| *** abregman has joined #openstack-infra | 20:11 | |
| *** Apoorva has quit IRC | 20:11 | |
| *** abregman is now known as abregman|afk | 20:11 | |
| *** mfedosin has quit IRC | 20:12 | |
| *** aeng has joined #openstack-infra | 20:13 | |
| *** ldnunes has quit IRC | 20:15 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge https://review.openstack.org/389127 | 20:15 |
| *** claudiub|2 has quit IRC | 20:16 | |
| *** Goneri has joined #openstack-infra | 20:16 | |
| *** mfedosin has joined #openstack-infra | 20:20 | |
| *** devkulkarni has quit IRC | 20:21 | |
| *** ccamacho has left #openstack-infra | 20:22 | |
| wznoinsk | hi infra, would someone have a moment to add me to https://review.openstack.org/#/admin/groups/1610,members ? | 20:22 |
| wznoinsk | or better yet, intel-nfv-ci account | 20:22 |
| *** thorst_ has quit IRC | 20:22 | |
| wznoinsk | "Intel NFV CI <openstack-nfv-ci@intel.com>" | 20:23 |
| pleia2 | wznoinsk: added you | 20:23 |
| *** ihrachys has quit IRC | 20:24 | |
| wznoinsk | pleia2, merci | 20:24 |
| *** ihrachys has joined #openstack-infra | 20:24 | |
| *** ijw_ has joined #openstack-infra | 20:25 | |
| *** vsaienko has quit IRC | 20:25 | |
| *** mdrabe has quit IRC | 20:25 | |
| *** mhickey has quit IRC | 20:26 | |
| *** flip214_ is now known as flip214 | 20:27 | |
| *** dave-mccowan has quit IRC | 20:28 | |
| *** ijw has quit IRC | 20:28 | |
| *** timello has quit IRC | 20:30 | |
| *** florianf has quit IRC | 20:30 | |
| *** maishsk has quit IRC | 20:32 | |
| *** maishsk has joined #openstack-infra | 20:33 | |
| *** Goneri has quit IRC | 20:33 | |
| *** jkilpatr has quit IRC | 20:33 | |
| *** timello has joined #openstack-infra | 20:36 | |
| *** ijw_ has quit IRC | 20:37 | |
| *** maishsk has quit IRC | 20:37 | |
| *** Rockyg has quit IRC | 20:38 | |
| *** mat128|afk is now known as mat128 | 20:43 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas tempest jobs for legacy and v1 voting https://review.openstack.org/389320 | 20:44 |
| *** john-davidge has quit IRC | 20:44 | |
| *** mat128 is now known as mat128|gone | 20:44 | |
| *** john-davidge has joined #openstack-infra | 20:44 | |
| *** askb has joined #openstack-infra | 20:47 | |
| *** dprince has quit IRC | 20:48 | |
| *** john-davidge has quit IRC | 20:49 | |
| *** tphummel has joined #openstack-infra | 20:50 | |
| *** stream10 has quit IRC | 20:53 | |
| *** dave-mccowan has joined #openstack-infra | 20:54 | |
| *** amitgandhinz has quit IRC | 20:54 | |
| *** priteau has quit IRC | 20:54 | |
| *** amitgandhinz has joined #openstack-infra | 20:55 | |
| *** mfedosin has quit IRC | 20:55 | |
| *** thorst_ has joined #openstack-infra | 20:56 | |
| *** jkilpatr has joined #openstack-infra | 20:59 | |
| *** tphummel has quit IRC | 21:00 | |
| *** ijw has joined #openstack-infra | 21:01 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/project-config: Bump entercloud max servers up from 0 https://review.openstack.org/389353 | 21:02 |
| pleia2 | clarkb: ^ | 21:02 |
| *** edmondsw has quit IRC | 21:02 | |
| clarkb | approved | 21:02 |
| *** raildo has quit IRC | 21:03 | |
| *** yolanda has quit IRC | 21:03 | |
| *** devkulkarni has joined #openstack-infra | 21:04 | |
| *** inc0 has quit IRC | 21:04 | |
| *** dave-mcc_ has joined #openstack-infra | 21:05 | |
| *** devkulkarni has quit IRC | 21:05 | |
| *** baoli_ has quit IRC | 21:05 | |
| *** srobert has quit IRC | 21:05 | |
| *** baoli has joined #openstack-infra | 21:05 | |
| *** ijw has quit IRC | 21:07 | |
| *** Goneri has joined #openstack-infra | 21:08 | |
| *** baoli_ has joined #openstack-infra | 21:08 | |
| *** baoli_ has quit IRC | 21:08 | |
| *** dave-mccowan has quit IRC | 21:08 | |
| *** baoli_ has joined #openstack-infra | 21:09 | |
| *** baoli has quit IRC | 21:10 | |
| *** trown is now known as trown|outtypewww | 21:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Bump entercloud max servers up from 0 https://review.openstack.org/389353 | 21:12 |
| mordred | clarkb: you got image uplodas to ecs to work I take it? | 21:13 |
| pleia2 | we did! | 21:13 |
| *** matrohon has joined #openstack-infra | 21:13 | |
| pleia2 | v1 api | 21:13 |
| clarkb | ya glance v1 ftw | 21:13 |
| mordred | pleia2: was the container_format relevant at all? | 21:13 |
| mordred | yay! | 21:13 |
| pleia2 | nope | 21:13 |
| mordred | ossum | 21:13 |
| clarkb | I +2ed oscc change | 21:14 |
| mordred | I didn't want to have to implement support for that :) | 21:14 |
| pleia2 | hah, right | 21:14 |
| pleia2 | it looks good though, 3 clouds, 7 regions :) | 21:14 |
| pleia2 | exciting | 21:14 |
| anteaya | might I inquire as to what ecs stands for? | 21:15 |
| mordred | woot! | 21:15 |
| mordred | anteaya: entercloudsuite | 21:15 |
| mordred | anteaya: it's a european openstack public cloud based in italy | 21:15 |
| anteaya | interesting | 21:15 |
| anteaya | how wonderful | 21:16 |
| mordred | clarkb: Unable to create new object: /home/gerrit2/review_site/git/openstack/os-client-config.git/objects/e9/21010db6806926f4a6e5ef451b1f6ca6df349c | 21:16 |
| *** gyee has joined #openstack-infra | 21:16 | |
| mordred | clarkb: I got that trying to rebase https://review.openstack.org/#/c/388199 | 21:16 |
| mordred | clarkb: any immediate thoughts? I know you were poking at other merge issues earlier | 21:16 |
| clarkb | that means jgit cant resolve the merge | 21:16 |
| clarkb | usually manual rebase fixes | 21:16 |
| mordred | cool | 21:16 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Add support for volumev3 service type https://review.openstack.org/389356 | 21:16 |
| clarkb | solums problem was even that didnt help | 21:17 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Clarify how to set SSL settings https://review.openstack.org/388199 | 21:17 |
| mordred | I have manually rebased | 21:17 |
| mordred | clarkb: also - apparently there is a volumev3 service type now | 21:17 |
| pabelanger | evening | 21:18 |
| mordred | it's a pabelanger ! | 21:18 |
| pabelanger | just catching up on backscroll, sudo issues from this morning was ansible config issue? | 21:18 |
| *** baoli_ has quit IRC | 21:19 | |
| *** r-mibu has quit IRC | 21:19 | |
| *** baoli has joined #openstack-infra | 21:19 | |
| *** r-mibu has joined #openstack-infra | 21:19 | |
| prometheanfire | for those wondering (I think mordred AJaeger dirk and pabelanger) gerrit doesn't know how to tell jgit to lower the context when merging patches | 21:19 |
| dirk | prometheanfire: thx | 21:20 |
| prometheanfire | yarp | 21:20 |
| mordred | prometheanfire: ah | 21:21 |
| jhesketh | Morning | 21:21 |
| *** zz_dimtruck is now known as dimtruck | 21:21 | |
| anteaya | morning jhesketh | 21:21 |
| prometheanfire | moin | 21:21 |
| pabelanger | looks that way: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2016-10-20T17:17:54 | 21:22 |
| pabelanger | TIL | 21:22 |
| mordred | it's a jhesketh ! | 21:22 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Remove validate_auth_ksc https://review.openstack.org/368778 | 21:23 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Fix a bunch of tests https://review.openstack.org/368776 | 21:23 |
| prometheanfire | pabelanger: ? | 21:23 |
| prometheanfire | pabelanger: sudo thing? | 21:24 |
| pabelanger | prometheanfire: there was an issue with ansible and sudo this morning, looks like we got the fix (ansible.cfg change) | 21:24 |
| prometheanfire | ah | 21:24 |
| *** esikachev has quit IRC | 21:24 | |
| pabelanger | was just checking to see if it was addressed | 21:24 |
| *** baoli_ has joined #openstack-infra | 21:25 | |
| *** baoli_ has quit IRC | 21:25 | |
| *** cody-somerville has quit IRC | 21:25 | |
| *** csomerville has joined #openstack-infra | 21:25 | |
| *** baoli has quit IRC | 21:26 | |
| jhesketh | mordred: there's been some gerrit restarts etc... anything left outstanding or are things ticking along again? | 21:26 |
| *** baoli_ has joined #openstack-infra | 21:26 | |
| *** tnovacik has quit IRC | 21:26 | |
| *** baoli_ has quit IRC | 21:27 | |
| *** baoli has joined #openstack-infra | 21:27 | |
| *** gordc has quit IRC | 21:28 | |
| *** matrohon has quit IRC | 21:29 | |
| *** baoli has quit IRC | 21:29 | |
| *** baoli has joined #openstack-infra | 21:30 | |
| *** cody-somerville has joined #openstack-infra | 21:30 | |
| *** cody-somerville has joined #openstack-infra | 21:30 | |
| mordred | jhesketh: things are in good shape at the moment | 21:30 |
| fungi | okay, i'm back and caught up on scrollback now | 21:31 |
| mordred | jhesketh: we also found the bug with the new v2.5 change that was causing sudo commands in release jobs to hang and that's rolled out | 21:31 |
| anteaya | fungi: welcome back | 21:31 |
| fungi | as for the stuck changes for solum, the only thing i know to do next is unset submitted for them in the gerrit db... i'm not sure if that requires a reindex or just a cache flush though | 21:31 |
| *** baoli has quit IRC | 21:32 | |
| jhesketh | mordred: that sounds nasty, but good job :-) | 21:32 |
| *** csomerville has quit IRC | 21:32 | |
| *** baoli has joined #openstack-infra | 21:32 | |
| *** vhosakot has quit IRC | 21:33 | |
| *** baoli has quit IRC | 21:33 | |
| clarkb | fungi: is there a way to resubmit without doing that? | 21:33 |
| clarkb | try to force it outside of zuul? | 21:33 |
| *** dave-mcc_ has quit IRC | 21:33 | |
| zaro | fungi: doesn't any change to db, not going thru gerrit, require a reindex? | 21:33 |
| *** Jeffrey4l has quit IRC | 21:34 | |
| zaro | clarkb: +1, restart the change. | 21:34 |
| fungi | zaro: not sure. for example, we make changes to the accounts and account_external_ids tables to resolve duplicate accounts and don't reindex | 21:34 |
| *** yamahata has quit IRC | 21:34 | |
| *** ijw has joined #openstack-infra | 21:34 | |
| fungi | i'm not entirely sure how to go about forcing it to redo the submit operation when gerrit already thinks it's submitted | 21:35 |
| *** matt-borland has quit IRC | 21:35 | |
| *** baoli has joined #openstack-infra | 21:35 | |
| zaro | is abadone and new change an option? | 21:35 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Update query for TLS bug 1630664 for neutron https://review.openstack.org/389360 | 21:36 |
| openstack | bug 1630664 in OpenStack Compute (nova) "Intermittent failure in n-api connecting to neutron to list ports after TLS was enabled in CI" [Medium,Confirmed] https://launchpad.net/bugs/1630664 | 21:36 |
| fungi | zaro: maybe, but it's a lot of changes apparently | 21:36 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Update query for TLS bug 1630664 for neutron https://review.openstack.org/389360 | 21:36 |
| anteaya | I missed the bit about why only solum is affected by this | 21:37 |
| *** baoli has quit IRC | 21:37 | |
| *** inc0 has joined #openstack-infra | 21:37 | |
| *** baoli_ has joined #openstack-infra | 21:37 | |
| fungi | well, half a dozen anyway | 21:37 |
| fungi | anteaya: no idea really, best guess is that gerrit got confused about something with the repo | 21:37 |
| fungi | https://review.openstack.org/#/q/project:openstack/solum+status:submitted | 21:37 |
| anteaya | how odd | 21:38 |
| *** baoli_ has quit IRC | 21:38 | |
| *** baoli has joined #openstack-infra | 21:39 | |
| *** baoli has quit IRC | 21:39 | |
| zaro | fungi, clarkb https://bugs.chromium.org/p/gerrit/issues/detail?id=600 | 21:40 |
| fungi | no new changes have merged since the 13th | 21:40 |
| fungi | for solum | 21:40 |
| zaro | dborowitz says try submit button or update db | 21:41 |
| fungi | zaro: slightly different. these aren't merged to the repo (or at least AJaeger checked and said they weren't) | 21:41 |
| *** baoli has joined #openstack-infra | 21:42 | |
| fungi | hrm, i've just spotted another inconsistency that may be related | 21:42 |
| mriedem | pretty please https://review.openstack.org/#/c/366933/ - we have a docs change fail in the gate b/c of the ceph job | 21:43 |
| mriedem | that shouldn't happen | 21:43 |
| *** baoli has quit IRC | 21:43 | |
| mordred | infra-root: could one of your +A this: https://review.openstack.org/#/c/389321/ | 21:43 |
| *** baoli has joined #openstack-infra | 21:43 | |
| *** claudiub|2 has joined #openstack-infra | 21:43 | |
| fungi | compare https://review.openstack.org/#/q/project:openstack/solum+status:merged+branch:master to http://git.openstack.org/cgit/openstack/solum/log/ and note that there's a change gerrit says has merged that doesn't appear in cgit | 21:44 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Update query for TLS bug 1630664 for neutron https://review.openstack.org/389360 | 21:44 |
| openstack | bug 1630664 in OpenStack Compute (nova) "Intermittent failure in n-api connecting to neutron to list ports after TLS was enabled in CI" [Medium,Confirmed] https://launchpad.net/bugs/1630664 | 21:44 |
| zaro | not merged? maybe just force it? | 21:44 |
| *** simondodsley has quit IRC | 21:44 | |
| fungi | yeah, i think the problem is that gerrit thinks it merged 385892 into the repo but it didn't | 21:44 |
| jhesketh | mordred: +w | 21:45 |
| fungi | so all the other changes it wants to merge are now stuck behind that | 21:45 |
| mordred | jhesketh: thanks! | 21:45 |
| zaro | fungi: try pressing the submit button? | 21:46 |
| fungi | zaro: on which change? | 21:46 |
| *** baoli has quit IRC | 21:46 | |
| *** baoli has joined #openstack-infra | 21:46 | |
| fungi | `git log master` in ~gerrit2/review_site/git/openstack/solum.git definitely doesn't include the commit for 385892 even though gerrit says that was the last one to merge to master | 21:46 |
| fungi | no, nevermind. that's stable/newton | 21:47 |
| zaro | all the merge pending ones. | 21:47 |
| fungi | so scratch that theory | 21:48 |
| *** baoli has quit IRC | 21:48 | |
| fungi | zaro: i'll elevate my privileges and see if it gives me a submit button, even though it claims it's already submitted | 21:48 |
| *** baoli has joined #openstack-infra | 21:49 | |
| zaro | ahh right. probably won't have the button then. can you do force push? | 21:50 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Treat yaml files as plain text https://review.openstack.org/389321 | 21:50 |
| zaro | fungi: https://groups.google.com/d/msg/repo-discuss/XdRmhvoW1MY/VBzLEQUu-8cJ | 21:51 |
| *** cody-somerville has quit IRC | 21:51 | |
| *** cody-somerville has joined #openstack-infra | 21:51 | |
| *** yamahata has joined #openstack-infra | 21:51 | |
| fungi | strange, even in project-bootstrappers, and with a force reload of teh tab, i only get code review -1..+1 if i try to review one of those changes | 21:51 |
| fungi | i'm going to try abandoning and restoring https://review.openstack.org/388334 | 21:54 |
| fungi | confirmed, abandoning and restoring got it back to a ready to submit state, but then the submit button put it back to submitted merge pending | 21:56 |
| *** thorst_ has quit IRC | 21:56 | |
| *** thorst_ has joined #openstack-infra | 21:57 | |
| fungi | org.eclipse.jgit.errors.ObjectWritingException: Unable to create new object: /home/gerrit2/review_site/git/openstack/solum.git/objects/16/56fbc8b649744c2785403dda480ee5dcc5baee | 21:57 |
| fungi | the backtrace looks similar to the one clarkb found | 21:58 |
| *** jordanP has joined #openstack-infra | 21:58 | |
| fungi | though looking in the logs, it's not the only one | 21:58 |
| clarkb | which should be just like the one that we get on a normal needs rebase | 21:58 |
| fungi | org.eclipse.jgit.errors.ObjectWritingException: Unable to create new object: /home/gerrit2/review_site/git/openstack/puppet-glance.git/objects/87/6d2b4b38f6a7007c4b652513b417e84887ff34 | 21:58 |
| fungi | though https://review.openstack.org/#/q/status:submitted is only showing solum changes, so that may be a benign one | 21:59 |
| clarkb | fungi: ya its a normal error for anything that can't merge | 22:00 |
| clarkb | fungi: thats how the merge checker works | 22:00 |
| fungi | ahh, the is:mergeable query seems to trigger some of these | 22:00 |
| zaro | turn up the logging for more info? | 22:00 |
| fungi | yeah, i was considering that as a next step | 22:00 |
| *** gildub has joined #openstack-infra | 22:03 | |
| *** baoli has quit IRC | 22:03 | |
| *** baoli has joined #openstack-infra | 22:03 | |
| *** thorst_ has quit IRC | 22:05 | |
| wznoinsk | hi infra, any chance someone allow me edit info on my barcelona ticket in eventbrite? | 22:07 |
| fungi | mriedem: i don't see where 366933 would stop docs changes from triggering ceph jobs unless i'm missing something. that change is totally about skipping jobs for the (frequent?) occurrence of adding lines to .gitignore | 22:07 |
| wznoinsk | currently it says: The organizer has elected to not make this information editable. Contact the organizer if you made a mistake. | 22:07 |
| *** jordanP has quit IRC | 22:07 | |
| clarkb | wznoinsk: like your name/employer etc? I think we can get an email address for who can help you with that | 22:07 |
| wznoinsk | clarkb, I saw 'edit' button this morning, see only the msg above now | 22:08 |
| wznoinsk | clarkb, (UTC morning) | 22:09 |
| clarkb | wznoinsk: summit@openstack.org | 22:09 |
| fungi | wznoinsk: it's possible they've locked it now because they're dumping the registration data into the systems for the venue | 22:09 |
| mriedem | fungi: that's true, it was a docs change for adding the policy sample to nova docs - and the policy sample was added to .gitignore | 22:09 |
| wznoinsk | clarkb, fungi roger | 22:09 |
| mriedem | so not a docs thing, but still | 22:09 |
| *** esikachev has joined #openstack-infra | 22:09 | |
| fungi | mriedem: fwiw, i count 4 total changes to that file this year. still really seems like over-optimizing | 22:11 |
| *** baoli_ has joined #openstack-infra | 22:12 | |
| *** baoli has quit IRC | 22:12 | |
| *** tykeal has left #openstack-infra | 22:13 | |
| mriedem | does it hurt anything? | 22:13 |
| mriedem | precious test nodes.... | 22:13 |
| fungi | i guess i shouldn't talk to nova people about review backlog, but... we have a pretty substantial review backlog (in addition to trying to fix things that are perpetually broken), so some up-front consideration to avoid changes that will marginally improve the efficiency of four changes out of tens of thousands or more a year is appreciated | 22:15 |
| *** baoli_ has quit IRC | 22:15 | |
| *** esikachev has quit IRC | 22:15 | |
| *** baoli has joined #openstack-infra | 22:16 | |
| mriedem | sorry, it's a 1 line change that was out there for awhile, i asked auggy to add that b/c we've seen things fail when they shouldn't, like i know the .gitreview change for stable/newton failed too when it shouldn't have | 22:16 |
| fungi | also, many of us would rather revert the skip-if support entirely (it's fiddly and causes far more problems than it fixes), so most of us just don't review chanegs that alter skip-if blocks because we don't want to be the ones responsible when they cause, say, half nova's jobs to cease being run and end up getting broken changes into the tree or whatever | 22:17 |
| *** r-daneel has quit IRC | 22:17 | |
| *** baoli_ has joined #openstack-infra | 22:18 | |
| *** baoli has quit IRC | 22:18 | |
| *** abregman|afk has quit IRC | 22:19 | |
| *** amitgandhinz has quit IRC | 22:19 | |
| fungi | the people who approved that feature have since regretted it | 22:19 |
| *** Goneri has quit IRC | 22:20 | |
| *** thorst_ has joined #openstack-infra | 22:20 | |
| mriedem | the negative logic in the skip-if thing sucks, but it seems pretty useful to not run big resource hog jobs like multinode grenade on docs/unit test changes | 22:20 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 22:21 |
| fungi | i believe the idea for the future is to have better per-project filtering mechanisms (as opposed to the file filter which is only per-job today so a bit clunky for jobs that run across multiple projects) | 22:21 |
| fungi | the rule implementation for skip-if though makes it far too easy to skip jobs you didn't mean to, or completely deadlock certain changes because there is no job that matches the files it touches | 22:22 |
| *** EricGonczer_ has quit IRC | 22:23 | |
| *** thorst_ has quit IRC | 22:24 | |
| fungi | zaro: clarkb: i'm going to `gerrit logging set-level debug com.google.gerrit.server.git` | 22:26 |
| fungi | then i'll do the abandon/restore/submit dance again and see if we get any more useful detail | 22:26 |
| clarkb | ok | 22:26 |
| *** rossella_s has quit IRC | 22:28 | |
| *** cardeois has quit IRC | 22:28 | |
| *** rossella_s has joined #openstack-infra | 22:28 | |
| fungi | zaro: is that supposed to increase logging detail in the error_log? i still just see the same old backtrace | 22:29 |
| zaro | it's supposed to. | 22:29 |
| zaro | get-log to see if it's set? | 22:30 |
| fungi | no, scratch that, i'm seeing an earlier timestamp on this backtrace | 22:30 |
| *** csomerville has joined #openstack-infra | 22:30 | |
| *** baoli_ has quit IRC | 22:32 | |
| *** baoli has joined #openstack-infra | 22:33 | |
| fungi | zaro: `gerrit logging ls-level` apparently, but yes it shows it's set | 22:33 |
| *** cody-somerville has quit IRC | 22:33 | |
| *** ijw has quit IRC | 22:33 | |
| *** baoli has quit IRC | 22:34 | |
| fungi | okay, i do see additional detail, like "Found 5 existing heads" and "Running submit strategy MergeIfNecessary for 6 commits" | 22:34 |
| *** baoli has joined #openstack-infra | 22:34 | |
| *** baoli has quit IRC | 22:35 | |
| *** baoli has joined #openstack-infra | 22:36 | |
| fungi | clarkb: zaro: http://paste.openstack.org/show/586636 | 22:36 |
| *** baoli has quit IRC | 22:36 | |
| fungi | why is it trying a 6-way merge there? | 22:37 |
| clarkb | huh | 22:37 |
| clarkb | what change did you abandon restore? | 22:37 |
| fungi | the change on which i submitted (388334) doesn't look like it has any related changes | 22:38 |
| *** baoli has joined #openstack-infra | 22:39 | |
| clarkb | and its not in that list either? | 22:39 |
| *** xarses has quit IRC | 22:39 | |
| fungi | right, this is getting strange... | 22:39 |
| clarkb | I wonder if this is anoptimization of merge if necessary to reduce merge commits? | 22:39 |
| *** SumitNaiksatam has quit IRC | 22:39 | |
| clarkb | basically pile together all the changes that can merge but makes conflicts more likely | 22:40 |
| *** baoli has quit IRC | 22:40 | |
| fungi | maybe if i abandon them all and restore just this one, then try to submit it again? | 22:40 |
| zaro | i'm confused. which change did you submit? | 22:40 |
| fungi | 388334 | 22:40 |
| clarkb | fungi: ya thatd what i am thinking if it is a bad optimization | 22:41 |
| zaro | it's 388334 dependent on that same topic list? why do you say it doesn't ahve any related changes? | 22:42 |
| *** baoli has joined #openstack-infra | 22:42 | |
| fungi | there are lots of changes in other projects with that topic. there are no changes in solum depending on that change, and its parent is already merged and in the repo | 22:42 |
| zaro | ahh, ok. i misread that panel again. | 22:43 |
| fungi | yeah, i misread it constantly | 22:43 |
| *** dtardivel has quit IRC | 22:44 | |
| fungi | "Opened branch refs/heads/master: commit 726a78929943c1d1bdaf4fc115cb79694fd79a1c" looks good at least. that's the master branch tip and its the parent of the 388334 change i attempted to submit | 22:45 |
| *** baoli has quit IRC | 22:46 | |
| zaro | can any other new change be sumbitted on this project? maybe the stuck changes is causing all changes to be stuck? | 22:47 |
| *** baoli has joined #openstack-infra | 22:48 | |
| fungi | maybe... the one in the backtrace at least is https://review.openstack.org/385893 | 22:48 |
| fungi | perhaps if i abandon that one temporarily then the others can merge? | 22:49 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/ansible-role-puppet: Update README with info about puppet apply https://review.openstack.org/333459 | 22:49 |
| fungi | that's also the one that was submitted earliest of them all, so it's possible | 22:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Allow ARA to test different versions of Ansible in the gate https://review.openstack.org/389384 | 22:50 |
| *** pahuang has joined #openstack-infra | 22:51 | |
| zaro | fungi: worth a shot but i'm guessing they all need to be unstuck before anymore changes will get merged | 22:52 |
| *** baoli has quit IRC | 22:52 | |
| zaro | try to abandone one, then try abandoning all stuck ones? | 22:52 |
| *** baoli_ has joined #openstack-infra | 22:53 | |
| *** marst has quit IRC | 22:53 | |
| asselin__ | does anyone know why this would fail? http://logs.openstack.org/75/360675/2/check/gate-ansible-role-cloud-launcher-dsvm-ansible-func-centos-7/e276ae5/console.html#_2016-10-20_22_38_34_397593 Collecting ansible==2.1.2.0 I see the package here: http://mirror.regionone.osic-cloud1.openstack.org/pypi/simple/ansible/ | 22:54 |
| fungi | zaro: yeah, that's next on the list. this was trying to submit 388334 again after abandoning 385893: http://paste.openstack.org/show/586637 | 22:54 |
| fungi | this time 388334 did show up in the list of what it was trying to merge at least | 22:55 |
| fungi | though the one which failed to merge this time per the backtrace was the commit for 384199 | 22:56 |
| *** baoli_ has quit IRC | 22:57 | |
| zaro | that one appeared on both list. so you gonna abandone one by one or all? | 22:57 |
| fungi | not sure yet. i know there are also just plain issues we've seen in the past with n-way merges and jgit | 22:58 |
| *** baoli has joined #openstack-infra | 22:58 | |
| fungi | so that could be manifesting here | 22:58 |
| zaro | wouldn't we see this same thing in other projects if that were true? | 23:00 |
| fungi | abandoned one more, submit still failed but now the number of changes it's attempting to merge has dropped to 4 | 23:00 |
| fungi | zaro: i'm not sure, i think something happened to get multiple changes into a submitted merge pending state, and now none can merge because they all end up in this state and cause issues for each other due to gerrit trying to merge them all when you ask to merge one (because they're all already submitted) | 23:01 |
| clarkb | we may see it in other cases but usually one merges then you tebase yhe others like mordred did earlier | 23:02 |
| clarkb | I wonder if it would do better serializing the merges | 23:02 |
| fungi | success! once i got the set down to 3 in a submitted merge pending state, i was able to get one to merge | 23:02 |
| *** baoli has quit IRC | 23:03 | |
| zaro | cool, good thing you went one at a time :) | 23:03 |
| fungi | in fact, supporting this theory, it merged the 3 remaining which were in that state all at the same time when i resubmitted one of them | 23:04 |
| fungi | i've restored the other 3 now | 23:04 |
| fungi | another possibility is that we somehow ended up with 2 changes in submitted merge pending which conflicted with one another, and getting one of them abandoned allowed the others to merge (just didn't know which one) | 23:05 |
| fungi | but i think that's less likely since i saw several different commits show up as unable to merge out of the set as i went through abandoning changes | 23:06 |
| clarkb | ya which would be the reason to serialize | 23:06 |
| *** gyee has quit IRC | 23:06 | |
| fungi | oh, though 385893 is showing up in proper merge conflict now | 23:06 |
| zaro | ohh nice! | 23:07 |
| *** rlandy has quit IRC | 23:08 | |
| fungi | eh, something strange is still afoot. i tried to submit 384199 since it didn't show in merge conflict and it's now submitted merge pending | 23:08 |
| *** ihrachys has quit IRC | 23:09 | |
| fungi | log says it was only trying to merge 1 commit at least | 23:09 |
| fungi | i'm still not sure what this line means: [openstack/solum,refs/heads/master@23:08:11]: Found 5 existing heads | 23:09 |
| fungi | why would refs/heads/master have 5 existing heads? | 23:10 |
| zaro | no clue either | 23:10 |
| clarkb | master newton mitaka kilo and something? | 23:10 |
| fungi | oh, maybe it's choosing master out of 5 available heads | 23:10 |
| clarkb | http://git.openstack.org/cgit/openstack/solum | 23:11 |
| clarkb | shows 5 branches/heads | 23:11 |
| fungi | cookiecutter, master, readme-start, stable/mitaka, stable/newton make 5, yeah | 23:11 |
| *** rbrndt has quit IRC | 23:11 | |
| fungi | on the assumption that 384199 needed a rebase even though it didn't claim a merge conflict, i've rebased it | 23:12 |
| fungi | i was able to get 387050 to merge successfully | 23:12 |
| fungi | and i've rebased 385893 to clear its merge conflict now | 23:13 |
| *** mriedem has quit IRC | 23:13 | |
| fungi | both ended up being trivial rebases | 23:13 |
| zaro | 384199 still says cannot merge? | 23:14 |
| openstackgerrit | Kevin Fox proposed openstack-infra/project-config: Add kolla-kubernetes multinode job & remove dead code https://review.openstack.org/389390 | 23:15 |
| fungi | nope, i rebased it and it needs workflow now | 23:16 |
| fungi | waiting to see if its ci jobs re-pass | 23:16 |
| fungi | same for 385893 | 23:16 |
| *** dimtruck is now known as zz_dimtruck | 23:16 | |
| zaro | fungi: yeah, i see needs workflow but why does 384199 say "Cannot merge"? | 23:18 |
| *** Sukhdev has quit IRC | 23:18 | |
| zaro | doesn't that mean it's detecting a conflict still? | 23:18 |
| *** sflanigan has joined #openstack-infra | 23:19 | |
| zaro | maybe it needs a manual rebase, from command line? | 23:21 |
| fungi | oh, that was off the side of my browser so i didn't notice it | 23:22 |
| *** hongbin has quit IRC | 23:22 | |
| *** dave-mccowan has joined #openstack-infra | 23:24 | |
| *** inc0 has quit IRC | 23:24 | |
| *** thorst_ has joined #openstack-infra | 23:24 | |
| fungi | anyway, i've externally rebased that one now | 23:25 |
| *** sdague has quit IRC | 23:25 | |
| *** gouthamr has quit IRC | 23:27 | |
| *** r-daneel has joined #openstack-infra | 23:28 | |
| *** maeker has quit IRC | 23:28 | |
| *** dizquierdo has quit IRC | 23:28 | |
| *** verdurin has quit IRC | 23:30 | |
| *** Jeffrey4l has joined #openstack-infra | 23:31 | |
| *** toabctl has quit IRC | 23:33 | |
| *** thorst_ has quit IRC | 23:33 | |
| *** r-daneel has quit IRC | 23:33 | |
| *** verdurin has joined #openstack-infra | 23:33 | |
| *** eharney has quit IRC | 23:35 | |
| *** markvoelker has quit IRC | 23:37 | |
| *** gildub has quit IRC | 23:37 | |
| *** dingyichen has joined #openstack-infra | 23:37 | |
| *** mountpoint has quit IRC | 23:39 | |
| *** Julien-zte has quit IRC | 23:39 | |
| *** toabctl has joined #openstack-infra | 23:44 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-bandersnatch: Add serveraliases to bandersnatch::mirror https://review.openstack.org/389393 | 23:45 |
| *** rhallisey has quit IRC | 23:45 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-bandersnatch: Add serveraliases to bandersnatch::mirror https://review.openstack.org/389393 | 23:47 |
| fungi | i've also reapproved (the rebased) 385893 to make sure it merges naturally on its own through the ci | 23:53 |
| prometheanfire | anyone know jeremy stanley on irc, dunno his nick | 23:53 |
| fungi | prometheanfire: nobody here but us chickens | 23:53 |
| fungi | er, i mean that would be me, yes | 23:53 |
| prometheanfire | oh lol | 23:54 |
| fungi | what did i do now? | 23:54 |
| *** gildub has joined #openstack-infra | 23:54 | |
| kfox1111 | fungi: got a quick sec to review this: https://review.openstack.org/#/c/389390/ ? | 23:59 |
| *** mriedem has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!