| *** ZZelle has quit IRC | 00:00 | |
| clarkb | the way openstack has been built to be a distributed system shouldn't rely on consistent gids. Definitely not keystone. Cinder and swift maybe | 00:00 |
|---|---|---|
| *** HeOS has quit IRC | 00:01 | |
| *** HeOS has joined #openstack-infra | 00:01 | |
| fungi | jhesketh: i'll defer to jeblair on interpreting the tc's "feedback" given they reviewed it at his urging, but i find the particular choice of domain to host it at more or less an implementation detail. have it publish to a directory on static.o.o and we can either have a separate vhost or we can add a line to the governance.o.o vhost. in the tc meeting they seemed to support the latter | 00:01 |
| *** davideagnello has quit IRC | 00:02 | |
| *** dizquierdo has quit IRC | 00:02 | |
| *** burgerk has joined #openstack-infra | 00:02 | |
| jhesketh | fungi: sure. In that case it sounds like it's ready for voting and we'll figure out the where in a bit | 00:03 |
| fungi | jhesketh: yep. i'd rather consider the choice of url a nonblocking detail and just put it up for vote as is | 00:03 |
| jhesketh | cheers | 00:03 |
| *** Sukhdev has quit IRC | 00:05 | |
| *** Sukhdev has joined #openstack-infra | 00:05 | |
| *** Sukhdev has quit IRC | 00:06 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Really reinstall requests on Fedora https://review.openstack.org/292651 | 00:07 |
| ianw | urgh, an hour of building and debugging to realise this stupid issue ^ | 00:07 |
| *** Sukhdev has joined #openstack-infra | 00:07 | |
| fungi | ouch | 00:07 |
| *** angdraug has quit IRC | 00:08 | |
| fungi | ianw: just --upgrade is insufficient | 00:08 |
| fungi | ? | 00:08 |
| fungi | i can't recall having to use --force-reinstall unless it thought the latest version was already installed | 00:09 |
| openstackgerrit | amrith proposed openstack/requirements: Update upper-constraints for python-troveclient v2.2.0 https://review.openstack.org/292652 | 00:09 |
| ianw | fungi: it thinks the latest version is installed (at this point, since the fedora packaged version == latest version) | 00:09 |
| ianw | it's all very silly, but see the comment above it | 00:09 |
| lifeless | if you're replacing a system package, you will need force-reinstall, *or* to remove the old package first | 00:10 |
| lifeless | the actual bug is on line 118 | 00:10 |
| lifeless | you're not removing the package metadata | 00:10 |
| lifeless | rm -rf .../requests-*.{egg,dist}-info | 00:11 |
| lifeless | IMO | 00:11 |
| lifeless | ianw: ^ | 00:11 |
| *** kencjohnston has joined #openstack-infra | 00:12 | |
| mordred | yah. I recommend removing the system package first - less pain | 00:12 |
| *** kencjohnston has quit IRC | 00:12 | |
| fungi | mordred: in this case i think there are base components of fedora which depend on python-requests | 00:12 |
| fungi | so removing the system package can be tricky | 00:12 |
| mordred | fungi: funny how this hole keeps resurfacing | 00:12 |
| ianw | yeah, we need the package installed so rpm doesn't re-install it and try to "unvendor" requests' sub-packages | 00:13 |
| mordred | of course, distro requests does the thing I want | 00:13 |
| ianw | i can remove the egg-info too, i guess. i think it's a different way to get to the same point | 00:14 |
| mordred | since it respects system CA | 00:14 |
| fungi | i only expect it to get worse over time as more and more basic components of distros get (re)written in python and don't want to reinvent certain wheels addressed by existing python libraries outside stdlib | 00:14 |
| mordred | too bad I have an installation script somewhere that actually copies the patched file from distro-packaged reuqests into place in a virtualenv | 00:14 |
| mordred | I mean | 00:14 |
| * mordred cries | 00:14 | |
| *** davideagnello has joined #openstack-infra | 00:14 | |
| morgan | mordred: *comfort* painful stuff. | 00:15 |
| * anteaya hands mordred a dry kitten | 00:15 | |
| morgan | fungi: it makes me sad everytime this comes up | 00:15 |
| anteaya | with a box of tissues | 00:15 |
| morgan | anteaya: i was wondering who had access to mordred's endless supply of <insert animal here>. | 00:15 |
| morgan | anteaya: ;) | 00:15 |
| ianw | mordred: well, it's probably really fedora-minimal's fault | 00:15 |
| anteaya | :D | 00:15 |
| fungi | morgan: he gets them from an open source | 00:16 |
| anteaya | ha ha ha | 00:16 |
| morgan | fungi: i see what you did there. | 00:16 |
| openstackgerrit | Tony Breeds proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes https://review.openstack.org/270273 | 00:16 |
| openstackgerrit | Tony Breeds proposed openstack-infra/release-tools: Actually use the URL specific in --project-list https://review.openstack.org/292657 | 00:16 |
| fungi | ianw: what is fedora-minimal doing to drag in requests? | 00:16 |
| lifeless | mordred: as fungi says, can't remove the system package, but we can remove the metadata more fully | 00:16 |
| openstackgerrit | Ken Johnston proposed openstack-infra/system-config: Add #osic to meetbot https://review.openstack.org/291798 | 00:17 |
| ianw | modred: the rpm'd packaged requests is fine, but it's not part of the minimal image. so puppet drags in the pip version at some point for something we install that wants it. later, devstack comes along and tries to install the packaged version | 00:17 |
| mordred | lifeless: yes indeed | 00:17 |
| fungi | ianw: a big part of why ubuntu-minimal exists is to not use cloud-init so we don't wind up with system packaged python-requests | 00:17 |
| * morgan waves at lifeless and goes back to lurking. | 00:17 | |
| mordred | ianw: why is devstack trying to install the _PACKAGED verions? | 00:17 |
| * lifeless waves morgan | 00:17 | |
| *** Qiming has joined #openstack-infra | 00:17 | |
| clarkb | fungi: and to handle rax network config | 00:17 |
| mordred | ianw: that sounds like a bug in devstack to me - distro requests is not going to be respecting upper-constraints :) | 00:17 |
| lifeless | mordred: because devstack does that on some platforms AIUI. | 00:17 |
| fungi | ianw: does fedora have a tool to tell you what the installed reverse-dependencies of a package are? | 00:17 |
| lifeless | mordred: which I think is nuts | 00:17 |
| mordred | lifeless: we should fix that | 00:17 |
| clarkb | we could cloud init in a venv for the first thing | 00:18 |
| *** eharney has quit IRC | 00:18 | |
| fungi | clarkb: good point. rackspace's nova-agent had similarly terrible dependencies we needed to avoid, one of which may also have been python-requests | 00:18 |
| mordred | clarkb: kindof. there are problems with cloud-init in a venv, but I believe we could figure it out | 00:18 |
| mordred | also, nova-agent likes to re-write files on running servers | 00:19 |
| ianw | mordred: cause something in http://logs.openstack.org/45/290445/2/check/gate-tempest-dsvm-platform-fedora23-nv/2c8150d/logs/devstacklog.txt.gz#_2016-03-14_18_34_50_919 pulls it in | 00:19 |
| clarkb | ya nova agent is pretty silly | 00:19 |
| *** burgerk has quit IRC | 00:19 | |
| fungi | ianw: if we can get at the reverse-depends we can track it back more easily to see what's actually dragging it in. that's how i tracked down a bunch of this on our ubuntu images over the years | 00:20 |
| ianw | mordred: i think the constraints will be correct, because we're "pip-ing" requests really. we just need to fool rpm that the package is installed so it doesn't try again | 00:20 |
| *** Kennan has quit IRC | 00:20 | |
| mordred | ianw: I disagree. we should stop installing the rpm | 00:21 |
| *** Kennan has joined #openstack-infra | 00:21 | |
| ianw | mordred: but what about when we need a package that depends on the rpm? | 00:21 |
| jeblair | fungi, jhesketh: in the reviews and moreso in the tc meeting, i think there was generally positive reaction to hosting at governance.o.o, and no opposition that i recall. so i think we should aim to implement it there. i leave the mechanics of that to fungi, but will be happy to draw up a quick amendment change now if we want to vote on them concurrently; or would be happy to push a new patchset over top if that is preferred. | 00:22 |
| *** Swami has quit IRC | 00:22 | |
| mordred | ianw: (my airplane wifi is being slow in loading that logfile - one sec while I read it) | 00:22 |
| fungi | jhesketh: i'll leave the choice between new patchset or follow-on amendment to you. for me they're basically the same outcome | 00:22 |
| jeblair | mordred is a spinning cursor | 00:22 |
| *** dims has joined #openstack-infra | 00:23 | |
| ianw | http://logs.openstack.org/20/289120/1/check/gate-tempest-dsvm-platform-fedora23-nv/429ff47/logs/devstacklog.txt.gz#_2016-03-07_03_12_09_191 is an actual log of python-requests being pulled in | 00:23 |
| mordred | ianw: do you have access to a box in the state after such a thing has happened? | 00:24 |
| fungi | ianw: i think for now the question is whether we _do_ need a package which depends on python-requests. usually those will be other python-based tools/libraries which can also be had from pypi | 00:24 |
| mordred | ianw: such that you can tell which of the packages depended on python-requests? | 00:24 |
| fungi | this is also why i was asking what tool on fedora tells me the installed reverse-dependencies of a given package. i know how to do it on debian derivatives but am pretty useless on rh-based distros these days | 00:25 |
| mordred | fungi: you could try "yum remove python-requests" and see what it suggests you need to uninstall too | 00:26 |
| fungi | oh, great point | 00:26 |
| ianw | i can get a system into this state, my cloud is just being a bit silly | 00:26 |
| fungi | though answering my own question, it looks like `repoquery --whatrequires ...` may be what i was looking for | 00:27 |
| fungi | something like `repoquery --whatrequires --installed --recursive python-requests` | 00:28 |
| openstackgerrit | Tony Breeds proposed openstack-infra/release-tools: DNM: Add a release tool to find bugs tagged for backport https://review.openstack.org/292660 | 00:28 |
| ianw | even still, i think *not* have python-requests leaves the system in a very precarious state that all sorts of devstack-plugins, etc, will find they screw themselves up by installing the packaged version over the pip version | 00:28 |
| *** thorst_ has quit IRC | 00:28 | |
| ianw | it usually doesn't matter that much -- it's just in this case with the vendoring rpm/cpio can't handle overwriting the symlinks to the vendored bits | 00:29 |
| fungi | ianw: why would they do that? we don't have that problem on ubuntu do we? | 00:29 |
| *** thorst_ has joined #openstack-infra | 00:29 | |
| ianw | maybe apt doesn't half-install the package? | 00:29 |
| ianw | or does the debian pkg not unvendor? or maybe it hard-links instead of symlinks and that works better? | 00:30 |
| fungi | ianw: i mean do we have a situation where devstack demands system packages which are depending on python-requests rather than pip installing anything it needs which uses requests? | 00:30 |
| *** Shrews has quit IRC | 00:31 | |
| *** docaedo has quit IRC | 00:31 | |
| fungi | i'm curious what is dragging the python-requests package into the fedora-minimal image in the hopes we might be able to avoid it | 00:31 |
| *** mnaser has quit IRC | 00:31 | |
| *** mordred has quit IRC | 00:31 | |
| *** phschwartz has quit IRC | 00:31 | |
| *** afazekas_ has quit IRC | 00:32 | |
| *** afazekas has joined #openstack-infra | 00:32 | |
| *** mordred has joined #openstack-infra | 00:32 | |
| *** phschwartz has joined #openstack-infra | 00:32 | |
| *** dims has quit IRC | 00:32 | |
| *** docaedo has joined #openstack-infra | 00:32 | |
| *** Shrews has joined #openstack-infra | 00:32 | |
| *** yamamoto_ has quit IRC | 00:34 | |
| ianw | fungi: the package, or the pip version? nothing drags the package in | 00:34 |
| jhesketh | if anybody has time to review https://review.openstack.org/#/c/289125/6 that'd be helpful :-) | 00:35 |
| fungi | ianw: you were saying the fedora-minimal element was probably responsible for causing the python-requests package to be installed on the system | 00:35 |
| fungi | ianw: so i'm curious how/why | 00:36 |
| ianw | fungi: no, it's responsible for it *not* being installed on the base system. which means that as part of system-config puppet, we pip install it. and then later devstack comes along and installs the rpm version, which then "half-installs" because it can't handle that it's trying to put symlinks where directories are | 00:36 |
| *** thorst_ has quit IRC | 00:37 | |
| fungi | ianw: i thought devstack was only using pypi for python libraries | 00:37 |
| fungi | didn't we make that switch a while back? | 00:38 |
| *** mnaser has joined #openstack-infra | 00:38 | |
| fungi | also we may want to fix whatever part of system-config is causing arbitrary python libraries to be globally preinstalled on our images | 00:38 |
| ianw | ok: http://paste.openstack.org/show/490445/ | 00:39 |
| fungi | i'm personally surprised we pip install requests while puppeting the images | 00:39 |
| fungi | except into virtualenvs | 00:39 |
| *** markvoelker has joined #openstack-infra | 00:40 | |
| *** ddieterly has joined #openstack-infra | 00:40 | |
| fungi | ianw: so euca2ools and/or gdisk are using python-requests i guess | 00:40 |
| fungi | gross, we're ending up with system-installed python-lxml too | 00:41 |
| *** Guest53749 has quit IRC | 00:41 | |
| ianw | fungi: yep. the python-lxml doesn't matter so much, i guess, because rpm can install it. if pip wants a different version, it would just re-install it over the system version | 00:43 |
| ianw | pip being whatever is in requirements | 00:43 |
| anteaya | jhesketh: +2 | 00:44 |
| jhesketh | anteaya: thank you :-) | 00:44 |
| anteaya | welcome :) | 00:44 |
| fungi | mordred: btw, your routine for fixing the ca trusts with pip installed requests is a function in devstack's lib/tls | 00:44 |
| mordred | fungi: :) | 00:44 |
| mordred | it's euca2ool | 00:46 |
| fungi | ianw: it looks like euca2ools is only being installed from system packages on rpm-based platforms? | 00:46 |
| fungi | i see it in files/rpms and files/rpms-suse but not for debian derivatives | 00:47 |
| fungi | are we not using it on ubuntu, or are we getting it there some other way? | 00:47 |
| ianw | fungi: i'll certainly investigate getting rid of that; i have some memory of us having old unit tests. i still want to make the change though, to avoid the system being in this precarious state and anyone else (plugins, in particular, installing essentially random stuff) having to debug this | 00:47 |
| mordred | https://review.openstack.org/292662 | 00:48 |
| ianw | not unit tests, exercises -- functional tests i guess | 00:48 |
| jhesketh | jeblair: ping | 00:48 |
| mordred | ianw: I think the design of desvstakc-gate nodes is currently suck that installing python from distro packages is unadvised and you get to keep both pieces when it breaks | 00:50 |
| mordred | ianw: now, I do think that us installing requests from pip not in a virtualenv in puppeting things is a bug and we should fix that | 00:50 |
| fungi | ianw: mordred looks like it was missed as a part of https://review.openstack.org/156594 | 00:50 |
| lifeless | I wonder if we can write a linter | 00:50 |
| amrith | heya folks in infra, anyone here got admin privileges to the ML server? I'm wondering if you could take a look at something for me ... ttx maybe? | 00:50 |
| lifeless | to catch things | 00:50 |
| jhesketh | jeblair: (sorry, just noticed your response above)... with using governance.o.o was there any opinion from the TC on how/where to link the elections from | 00:50 |
| anteaya | amrith: ttx is asleep | 00:50 |
| anteaya | amrith: just share the details you shared in -dev | 00:51 |
| anteaya | if someone has permissions and the time to help, they will say so | 00:51 |
| fungi | amrith: depends on which admin privs you need. list configuration or systems administration | 00:51 |
| mordred | ianw: but I agree that puppet installing requests with pip is a bug | 00:51 |
| mordred | ZOMG. My internet is so bad it's like watching computers in the 80s try to keep | 00:51 |
| mordred | up with touch typing | 00:51 |
| lifeless | mordred: they kept up fine | 00:51 |
| mordred | The duplex is killing it. I'm typing lines in another buffer and pasting them no | 00:52 |
| mordred | w | 00:52 |
| lifeless | mordred: until that newfangled windows thing came along and messed it up | 00:52 |
| amrith | fungi, here's my issue. I didn't get one (just one as far as I can tell) email that went o the ML today. It was this one http://markmail.org/message/43pfbo3oagjvo3lo | 00:52 |
| mordred | lifeless: actually - I remember my dad WAY overtyping the dos-based word processor | 00:52 |
| fungi | i do recall being able to type faster than my 110 baud acoustic coupled modem could send | 00:52 |
| amrith | so, others in my company got a copy, just one person didn't. | 00:52 |
| mordred | and also the old modems :) | 00:52 |
| lifeless | mordred: hmm, possible I guess. certainly modems had to buffer | 00:52 |
| amrith | so I checked on our mail server (I'm the admin there) and the mail trace shows that I didn't get a copy. | 00:52 |
| amrith | so I'm wondering why, I'm still on openstack-dev | 00:53 |
| amrith | I got other mails, before and after. | 00:53 |
| fungi | amrith: i'm checking the mta log on our end now | 00:53 |
| amrith | so was wondering if someone could look at the log. | 00:53 |
| amrith | thanks fungi | 00:53 |
| mordred | lifeless: my friend nathan's dad also overtyped the computer's ability to keep up. we were both in awe of the typing speeds of our respective dads at the time | 00:54 |
| anteaya | your dad's could type? | 00:55 |
| anteaya | dads | 00:55 |
| fungi | i wish i knew an easier way to get the message-id of a ml post out of pipermail than to download mailman's mbox archive | 00:55 |
| *** sdake has joined #openstack-infra | 00:55 | |
| anteaya | :( | 00:55 |
| clarkb | I recall doing this once | 00:55 |
| clarkb | let me check my command history | 00:55 |
| lifeless | mordred: apple ]['s? or trs-80s? or something else? | 00:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Really reinstall requests on Fedora https://review.openstack.org/292651 | 00:56 |
| clarkb | oh wait don't have my key loaded anymore | 00:56 |
| clarkb | derp | 00:56 |
| anteaya | :( | 00:56 |
| clarkb | fungi: ^ check my command history on lists as I seem to recall doing this when we were getting the spams | 00:56 |
| ianw | lifeless: ^ thanks, that version clears metadata f | 00:56 |
| mordred | anteaya: very well | 00:56 |
| *** thorst_ has joined #openstack-infra | 00:56 | |
| *** dims has joined #openstack-infra | 00:56 | |
| anteaya | wow | 00:56 |
| fungi | clarkb: for finding the full message headers of a ml post? | 00:57 |
| clarkb | fungi: just the id | 00:57 |
| *** Qiming has quit IRC | 00:57 | |
| *** ddieterly has quit IRC | 00:57 | |
| fungi | ahh | 00:57 |
| anteaya | my head is not dealing with the concept of a typing father very well at all | 00:57 |
| clarkb | fungi: its logged somewhere iirc | 00:57 |
| amrith | I can get you a msg-id | 00:57 |
| amrith | one second | 00:57 |
| amrith | 56E7117E.9050608@linux.vnet.ibm.com | 00:57 |
| *** esikachev has joined #openstack-infra | 00:58 | |
| *** dimtruck is now known as zz_dimtruck | 00:59 | |
| *** noorul has joined #openstack-infra | 00:59 | |
| ianw | fungi: so euca2ools isn't available via pypi, and i think it was probably really mistakenly removed as part of that purge. devstacks tools/euca.sh test would be broken there ... but it probably shows that it's irrelevant | 00:59 |
| *** Shrews has quit IRC | 00:59 | |
| *** ddieterly has joined #openstack-infra | 00:59 | |
| noorul | I am using gerrit customized by openstack ci team | 00:59 |
| noorul | But I don't see the workflow step | 00:59 |
| mordred | anteaya: well, my dad was a telco engineer, his dad was a doctor - both could program | 01:00 |
| anteaya | mordred: oh makes sense | 01:00 |
| anteaya | my dad could not program | 01:00 |
| fungi | ianw: since we removed it from being installed on ubuntu, i think we would have noticed if that were still being exercised | 01:00 |
| anteaya | he could build houses and chop wood though | 01:00 |
| *** bpokorny_ has quit IRC | 01:00 | |
| mordred | ianw: since ec2 is a plugin now, I do not think there is any reason for base devstack to install anything related to AWS | 01:00 |
| noorul | Am I missing something? | 01:01 |
| anteaya | noorul: I don't understand what you mean by the workflow step? | 01:01 |
| fungi | amrith: ftr, Message-ID: <56E7117E.9050608@linux.vnet.ibm.com> | 01:01 |
| amrith | fungi, but I did get this one ... <56E706FD.5070402@linux.vnet.ibm.com> | 01:01 |
| noorul | anteaya: Merge step | 01:01 |
| amrith | fungi, I sent you a screen shot of the message in question | 01:01 |
| fungi | amrith: ftr, Message-ID: <56E7117E.9050608@linux.vnet.ibm.com> | 01:01 |
| noorul | anteaya: Where in we put Workflow label +! | 01:01 |
| noorul | +1 | 01:01 |
| fungi | amrith: er, sorry for echoing twice | 01:02 |
| anteaya | noorul: commenting on patches | 01:02 |
| ianw | fungi: right, we don't run the exercises in the gate. but from a devstack point of view, you should be able to run the exercises. which i guess means the euca test is basically irrelevant because nobody is reporting issues with this | 01:02 |
| anteaya | noorul: you are at the step where you want to leave a comment on a patch in the ci-sandbox repo in gerrit? | 01:02 |
| ianw | mordred: yes, i guess it doesn't even work with the tools | 01:03 |
| anteaya | noorul: is this for a third party ci to comment on our gerrit's patches? | 01:03 |
| noorul | anteaya: I am in step where I want to merge the patch | 01:03 |
| ianw | because no plugin is installed. i can clean this up | 01:03 |
| amrith | yes, I posted them above as well ;) | 01:03 |
| noorul | anteaya: No, I am running this for non-openstack projects | 01:03 |
| amrith | I've sent you some screen shots by PM | 01:03 |
| anteaya | noorul: oh okay | 01:03 |
| amrith | notice that I get one, but not the other | 01:03 |
| noorul | anteaya: Trying to get a demo setup up | 01:03 |
| amrith | odd | 01:03 |
| *** esikachev has quit IRC | 01:03 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Publish election data to governance.o.o https://review.openstack.org/292666 | 01:03 |
| jhesketh | fungi: went with ^ to avoid losing existing votes | 01:03 |
| clarkb | noorul: you have to configure it in your all-projects iirc | 01:04 |
| anteaya | and I do believe I should be off to bed | 01:04 |
| anteaya | as I have to be up in several hours | 01:04 |
| anteaya | night all | 01:05 |
| amrith | g'night | 01:05 |
| anteaya | night | 01:05 |
| noorul | clarkb: I see, http://docs.openstack.org/infra/system-config/gerrit.html | 01:06 |
| clarkb | noorul: towards the bottom of http://docs.openstack.org/infra/system-config/gerrit.html#access-controls you will see where workflow is configured | 01:07 |
| *** sdake has quit IRC | 01:10 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Publish election data to governance.o.o https://review.openstack.org/292666 | 01:10 |
| fungi | amrith: it will take me a while to track this down since i need to correlate the message ids (which only seem to be logged with the sender details) to the deliveries. they're keyed by exim id values i just need to work up a map to find which copy of that message was destined to your address | 01:10 |
| amrith | thanks fungi ... I'm also opening a support request with our mail provider with the message traces that I have. | 01:11 |
| *** thorst_ has quit IRC | 01:12 | |
| *** zz_dimtruck is now known as dimtruck | 01:12 | |
| *** thorst_ has joined #openstack-infra | 01:13 | |
| *** markvoelker has quit IRC | 01:14 | |
| *** Sukhdev has quit IRC | 01:17 | |
| fungi | amrith: from what i can find in our mta logs, it doesn't appear mailman attempted to deliver a copy of that particular message to you | 01:17 |
| fungi | amrith: is it possible you have your account set to only subscribe to a limited set of tags on that list? | 01:18 |
| *** pvaneck has quit IRC | 01:18 | |
| amrith | fungi, I have my account set to send me stuff even if it is not tags that I've subscribed to | 01:18 |
| amrith | fungi, "Do you want to receive messages that do not match any topic filter?": Yes | 01:20 |
| *** thorst_ has quit IRC | 01:21 | |
| *** zhurong_ has joined #openstack-infra | 01:21 | |
| fungi | amrith: i believe that does not also send you things which do match a topic filter for a topic to which you are not subscribed | 01:22 |
| amrith | But this line (below it) does. "If no topics of interest are selected above, then you will receive every message sent to the mailing list. " | 01:22 |
| amrith | I chose no topics | 01:22 |
| fungi | okay, so you don't have any of them checked | 01:22 |
| amrith | in any event, stable isn't a valid topic on that list. | 01:22 |
| amrith | yes, none are checked. | 01:22 |
| *** gongysh has quit IRC | 01:24 | |
| *** tzn has joined #openstack-infra | 01:24 | |
| fungi | amrith: i agree, it looks like stable is not an official topic | 01:25 |
| fungi | i definitely see exim delivering messages to you when mailman asks it to do so, and i don't see any of them rejected | 01:25 |
| fungi | i also don't see mailman receiving any bounces for your address | 01:25 |
| fungi | i'm trying to figure out where and how it would log a decision not to send you a particular message though | 01:26 |
| *** chlong has quit IRC | 01:26 | |
| amrith | fungi, thanks for looking. I'm not sure it makes sense to take more time about this, I'll think about it again tomorrow and see what I can do. | 01:27 |
| amrith | thx for your time | 01:28 |
| *** tzn has quit IRC | 01:29 | |
| fungi | amrith: yeah, i'm coming up dry but maybe when jeblair's around i can get his eyes on it too. i think he's spent more time staring into the abyss of mailman logs than i have | 01:29 |
| * amrith shudders ... I was mail manager for a while, sendmail and I were on first name terms. much scotch was consumed ;) | 01:30 | |
| fungi | yeah, our mta (exim) i'm quite at home with, but the listserv (mailman) is a bit more opaque to me | 01:31 |
| Clint | sendmail was fun before spam becamse so prevalent | 01:31 |
| amrith | Clint, yes, those were the good days ;) | 01:31 |
| clarkb | think on the bright side, your experience with sendmail probably makes you a Make ninja | 01:31 |
| fungi | sendmail was fun back before people started using m4 macros to autogenerate the c object file they configured it with ;) | 01:32 |
| Clint | ha | 01:32 |
| amrith | yes, but today people have tools to generate makefiles and sendmail.cf ;) I used to do it by hand | 01:32 |
| *** wei-wang has joined #openstack-infra | 01:33 | |
| amrith | ah, what fungi said ... | 01:33 |
| amrith | time to go. have some more reviews to do before I sleep. | 01:34 |
| amrith | thanks fungi Clint ... g'night. | 01:34 |
| fungi | night amrith | 01:34 |
| *** tongli has quit IRC | 01:34 | |
| *** yamamoto_ has joined #openstack-infra | 01:34 | |
| *** kencjohnston_ has joined #openstack-infra | 01:36 | |
| *** woodster_ has quit IRC | 01:37 | |
| *** sridhar_ram has quit IRC | 01:39 | |
| *** zhurong_ has quit IRC | 01:39 | |
| *** burgerk has joined #openstack-infra | 01:39 | |
| *** burgerk has quit IRC | 01:39 | |
| *** rfolco has joined #openstack-infra | 01:40 | |
| *** zhurong_ has joined #openstack-infra | 01:41 | |
| *** kencjohnston_ has quit IRC | 01:42 | |
| *** yamamoto_ has quit IRC | 01:43 | |
| *** gildub has joined #openstack-infra | 01:43 | |
| *** tsg has quit IRC | 01:46 | |
| *** Qiming has joined #openstack-infra | 01:47 | |
| *** amrith is now known as _amrith_ | 01:49 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/puppet-httpd: Add support for alias and directory variables https://review.openstack.org/292674 | 01:50 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Add election alias to governance.openstack.org https://review.openstack.org/292675 | 01:50 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Publish election data to static server https://review.openstack.org/292676 | 01:50 |
| *** sbalukoff has quit IRC | 01:52 | |
| jhesketh | fungi, jeblair: ^ there's my first iteration of the election stuff (granted prematurely before the spec) | 01:54 |
| *** rockyg has joined #openstack-infra | 01:54 | |
| *** IlyaG has quit IRC | 01:57 | |
| *** woodster_ has joined #openstack-infra | 01:57 | |
| *** apoorvad has quit IRC | 01:58 | |
| *** piet has joined #openstack-infra | 02:01 | |
| *** salv-orlando has joined #openstack-infra | 02:02 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:03 | |
| *** salv-orl_ has quit IRC | 02:04 | |
| *** ddieterly has quit IRC | 02:05 | |
| *** rhallisey has quit IRC | 02:05 | |
| *** piet has quit IRC | 02:06 | |
| *** piet has joined #openstack-infra | 02:06 | |
| *** yamamoto has joined #openstack-infra | 02:07 | |
| *** rhallisey has joined #openstack-infra | 02:08 | |
| jeblair | jhesketh: i don't recall a preference from the TC as to path | 02:08 |
| *** gongysh has joined #openstack-infra | 02:09 | |
| jeblair | fungi, _amrith_: i don't have a lot of experience debugging mailman topics, sorry. happy to help as i can, but no magic bullets here | 02:10 |
| *** fawadkhaliq has quit IRC | 02:12 | |
| fungi | jeblair: sounds like it wasn't topic-related | 02:12 |
| jeblair | fungi: oh, i must have mis-skimmed? | 02:13 |
| fungi | jeblair: just that mailman (from what i can tell correlating the message id in the exim logs) never submitted a delivery to him for that particular message | 02:13 |
| fungi | i went scrounging around in mailman's various logs but couldn't find anything which would elucidate the matter at hand | 02:13 |
| jeblair | fungi: so, reason unknown, but no reason to suspect it was topics? | 02:13 |
| fungi | jeblair: according to amrith he had no topics selected in his mailman prefs for openstack-dev at all, so it should default to sending all messages | 02:14 |
| fungi | and also, the one subject tag which was used in the message wasn't an official subscribable list tag anyway | 02:14 |
| fungi | so shouldn't have been handled any differently than tagless messages afaik | 02:15 |
| fungi | so i'm fairly stumped | 02:15 |
| *** tphummel has quit IRC | 02:17 | |
| jhesketh | jeblair: ah okay. fungi said (above) that the tc seemed to prefer having it on g.o.o (noting that isn't a decision) | 02:21 |
| jhesketh | either way, I think it is indeed a better fit | 02:21 |
| jhesketh | but will update the commit messages to not lie | 02:21 |
| *** noorul has quit IRC | 02:21 | |
| *** gildub has quit IRC | 02:22 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Publish election data to governance.o.o https://review.openstack.org/292666 | 02:22 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Add election alias to governance.openstack.org https://review.openstack.org/292675 | 02:26 |
| *** yamamoto has quit IRC | 02:27 | |
| *** rockyg has quit IRC | 02:31 | |
| *** rfolco has quit IRC | 02:35 | |
| *** cody-somerville has joined #openstack-infra | 02:35 | |
| *** rhallisey has quit IRC | 02:35 | |
| *** dims has quit IRC | 02:38 | |
| *** armax has quit IRC | 02:38 | |
| *** cj has joined #openstack-infra | 02:39 | |
| *** dimtruck is now known as zz_dimtruck | 02:39 | |
| *** gongysh has quit IRC | 02:41 | |
| *** bryan_att has quit IRC | 02:42 | |
| *** tqtran has quit IRC | 02:42 | |
| *** kencjohnston_ has joined #openstack-infra | 02:48 | |
| *** kencjohnston_ has quit IRC | 02:49 | |
| *** IlyaG has joined #openstack-infra | 02:50 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting https://review.openstack.org/292687 | 02:51 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: WIP add element to install glean.rs https://review.openstack.org/292689 | 02:54 |
| mordred | fungi: https://review.openstack.org/292690 | 02:55 |
| mordred | fungi: just for you | 02:55 |
| mordred | jeblair: (although you might enjoy it too) | 02:55 |
| *** baoli_ has joined #openstack-infra | 02:57 | |
| *** asselin_ has joined #openstack-infra | 02:57 | |
| *** armax has joined #openstack-infra | 02:58 | |
| *** esikachev has joined #openstack-infra | 02:59 | |
| *** gildub has joined #openstack-infra | 03:01 | |
| *** HeOS has quit IRC | 03:02 | |
| *** HeOS has joined #openstack-infra | 03:02 | |
| *** yamahata has joined #openstack-infra | 03:02 | |
| *** doug-fish has joined #openstack-infra | 03:03 | |
| *** IlyaG has quit IRC | 03:03 | |
| *** esikachev has quit IRC | 03:03 | |
| *** fawadkhaliq has joined #openstack-infra | 03:06 | |
| *** yamamoto has joined #openstack-infra | 03:09 | |
| *** kushal has joined #openstack-infra | 03:11 | |
| *** markvoelker has joined #openstack-infra | 03:11 | |
| *** esker has quit IRC | 03:13 | |
| *** mtanino has quit IRC | 03:17 | |
| *** Qiming_ has joined #openstack-infra | 03:20 | |
| *** yuanying has quit IRC | 03:22 | |
| *** Kennan has quit IRC | 03:23 | |
| *** asettle has quit IRC | 03:23 | |
| *** Qiming has quit IRC | 03:24 | |
| *** Qiming__ has joined #openstack-infra | 03:25 | |
| *** Kennan has joined #openstack-infra | 03:26 | |
| *** erlon has quit IRC | 03:27 | |
| *** Qiming_ has quit IRC | 03:29 | |
| *** chlong has joined #openstack-infra | 03:30 | |
| fungi | mordred: an amusing opportunity to recount some relevant early history of our development! | 03:30 |
| *** kaisers has quit IRC | 03:36 | |
| *** links has joined #openstack-infra | 03:38 | |
| *** yuanying has joined #openstack-infra | 03:38 | |
| *** markvoelker has quit IRC | 03:44 | |
| *** bpokorny has joined #openstack-infra | 03:44 | |
| *** yuanying has quit IRC | 03:45 | |
| *** tphummel has joined #openstack-infra | 03:46 | |
| *** kaisers has joined #openstack-infra | 03:50 | |
| *** tphummel has quit IRC | 03:50 | |
| mordred | fungi: :) | 03:53 |
| mordred | fungi: that's kinda how I was looking at it | 03:54 |
| *** ayoung has quit IRC | 03:57 | |
| *** HeOS has quit IRC | 04:02 | |
| tonyb | mordred: ;p | 04:02 |
| *** HeOS has joined #openstack-infra | 04:03 | |
| *** IlyaG has joined #openstack-infra | 04:04 | |
| *** pahuang has quit IRC | 04:04 | |
| *** keedya has quit IRC | 04:05 | |
| *** sdake has joined #openstack-infra | 04:06 | |
| *** yuanying has joined #openstack-infra | 04:06 | |
| *** rguillebert has quit IRC | 04:07 | |
| *** ajmiller has joined #openstack-infra | 04:08 | |
| *** sdake_ has joined #openstack-infra | 04:09 | |
| *** IlyaG has quit IRC | 04:09 | |
| *** ajmiller_ has joined #openstack-infra | 04:09 | |
| *** sdake has quit IRC | 04:11 | |
| *** ajmiller has quit IRC | 04:13 | |
| *** bpokorny has quit IRC | 04:14 | |
| *** HeOS has quit IRC | 04:15 | |
| lifeless | mordred: -lol- | 04:17 |
| *** amotoki has joined #openstack-infra | 04:17 | |
| *** piet has quit IRC | 04:19 | |
| *** sdake_ has quit IRC | 04:19 | |
| *** piet has joined #openstack-infra | 04:19 | |
| *** armax has quit IRC | 04:20 | |
| *** pahuang has joined #openstack-infra | 04:21 | |
| *** ajmiller_ has quit IRC | 04:21 | |
| ianw | fungi / mordred : so, https://review.openstack.org/#/c/292651 (re-install requests really) i'd appreciate it if we could fix quickly, as we are currently shipping a completely broken requests due to not re-installing it. when the ec2 stuff goes through, i can think about removing this work-around | 04:24 |
| *** armax has joined #openstack-infra | 04:25 | |
| mordred | ianw: wfm | 04:26 |
| ianw | thanks, if you can +A it, i can feel good that it will be in tomorrows builds :) | 04:30 |
| *** piet has quit IRC | 04:30 | |
| ianw | which hopefully will upload, but that's another issue :) | 04:30 |
| *** piet has joined #openstack-infra | 04:31 | |
| *** treaki_ has joined #openstack-infra | 04:31 | |
| *** treaki__ has quit IRC | 04:35 | |
| *** sbalukoff has joined #openstack-infra | 04:42 | |
| *** wei-wang has quit IRC | 04:43 | |
| *** piet has quit IRC | 04:44 | |
| *** Sukhdev has joined #openstack-infra | 04:45 | |
| *** hichihara has joined #openstack-infra | 04:47 | |
| *** baoli_ has quit IRC | 04:50 | |
| ianw | fungi / mordred : ^ | 04:53 |
| *** piet has joined #openstack-infra | 04:53 | |
| *** woodster_ has quit IRC | 04:57 | |
| *** wei-wang has joined #openstack-infra | 04:59 | |
| *** tzn has joined #openstack-infra | 05:01 | |
| *** tzn has quit IRC | 05:01 | |
| openstackgerrit | Merged openstack-infra/system-config: Really reinstall requests on Fedora https://review.openstack.org/292651 | 05:04 |
| *** pahuang has quit IRC | 05:12 | |
| *** sdake has joined #openstack-infra | 05:15 | |
| *** amotoki has quit IRC | 05:21 | |
| *** roxanagh_ has quit IRC | 05:23 | |
| *** pahuang has joined #openstack-infra | 05:24 | |
| *** roxanagh_ has joined #openstack-infra | 05:24 | |
| *** armax has quit IRC | 05:29 | |
| *** HeOS has joined #openstack-infra | 05:30 | |
| *** amotoki has joined #openstack-infra | 05:31 | |
| *** sdake has quit IRC | 05:33 | |
| *** amotoki has quit IRC | 05:35 | |
| *** _nadya_ has joined #openstack-infra | 05:37 | |
| *** roxanagh_ has quit IRC | 05:40 | |
| *** roxanagh_ has joined #openstack-infra | 05:41 | |
| *** markvoelker has joined #openstack-infra | 05:41 | |
| *** Sukhdev has quit IRC | 05:43 | |
| *** piet has quit IRC | 05:44 | |
| *** roxanagh_ has quit IRC | 05:45 | |
| *** amotoki has joined #openstack-infra | 05:47 | |
| *** armax has joined #openstack-infra | 05:47 | |
| *** zzzeek has quit IRC | 05:49 | |
| *** abregman has joined #openstack-infra | 05:49 | |
| *** zzzeek has joined #openstack-infra | 05:50 | |
| *** salv-orl_ has joined #openstack-infra | 05:54 | |
| *** amotoki has quit IRC | 05:55 | |
| *** jaosorior has joined #openstack-infra | 05:57 | |
| *** salv-orlando has quit IRC | 05:57 | |
| *** kushal has quit IRC | 05:58 | |
| *** rlandy has quit IRC | 05:58 | |
| *** esikachev has joined #openstack-infra | 05:59 | |
| *** _nadya_ has quit IRC | 06:03 | |
| *** esikachev has quit IRC | 06:04 | |
| *** Daisy has joined #openstack-infra | 06:07 | |
| *** Daisy has quit IRC | 06:10 | |
| *** Daisy has joined #openstack-infra | 06:11 | |
| *** kushal has joined #openstack-infra | 06:14 | |
| *** markvoelker has quit IRC | 06:14 | |
| *** Daisy_ has joined #openstack-infra | 06:16 | |
| *** Daisy has quit IRC | 06:19 | |
| *** IlyaG has joined #openstack-infra | 06:22 | |
| *** IlyaG has quit IRC | 06:27 | |
| *** MarkAtwood has quit IRC | 06:33 | |
| *** jcooley has quit IRC | 06:35 | |
| *** asettle has joined #openstack-infra | 06:38 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:41 |
| *** roxanagh_ has joined #openstack-infra | 06:42 | |
| *** asettle has quit IRC | 06:43 | |
| *** placko_ has joined #openstack-infra | 06:46 | |
| *** Daisy_ has quit IRC | 06:49 | |
| *** Daisy has joined #openstack-infra | 06:50 | |
| *** roxanagh_ has quit IRC | 06:51 | |
| *** Daisy_ has joined #openstack-infra | 06:52 | |
| *** Daisy has quit IRC | 06:52 | |
| *** ildikov_ has quit IRC | 06:53 | |
| *** asettle has joined #openstack-infra | 06:54 | |
| *** HeOS has quit IRC | 07:01 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Don't take into account commit message for skip-if filter https://review.openstack.org/247383 | 07:01 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Split pipeline description by double newlines on status page https://review.openstack.org/239943 | 07:01 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Add possibility to use SSH instead of HTTP to get ref list https://review.openstack.org/246433 | 07:01 |
| *** HeOS has joined #openstack-infra | 07:01 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Report job status 'QUEUED' for queued jobs https://review.openstack.org/272967 | 07:02 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names https://review.openstack.org/282660 | 07:02 |
| *** oanson has quit IRC | 07:02 | |
| *** baoli has joined #openstack-infra | 07:02 | |
| *** baoli has quit IRC | 07:07 | |
| *** mikelk has joined #openstack-infra | 07:10 | |
| *** tesseract has joined #openstack-infra | 07:13 | |
| *** tesseract is now known as Guest97519 | 07:13 | |
| *** Daisy_ has quit IRC | 07:15 | |
| *** ihrachys has joined #openstack-infra | 07:17 | |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: move documents and python 2.7 checks to community https://review.openstack.org/292746 | 07:19 |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: move documents and python 2.7 checks to community https://review.openstack.org/292746 | 07:21 |
| lennyb | asselin: regarding firewall you are correct. I've checked this issue with our IT department and we suppose to be fine with the IP change. Thank you. | 07:22 |
| *** dingyichen has quit IRC | 07:27 | |
| *** maishsk has joined #openstack-infra | 07:29 | |
| *** fawadkhaliq has quit IRC | 07:30 | |
| *** scheuran has joined #openstack-infra | 07:33 | |
| yolanda | good morning | 07:41 |
| *** Daisy has joined #openstack-infra | 07:43 | |
| *** kingia_ has joined #openstack-infra | 07:46 | |
| *** placko_ has quit IRC | 07:47 | |
| *** kingia has quit IRC | 07:48 | |
| *** yamahata has quit IRC | 07:52 | |
| *** nmagnezi has joined #openstack-infra | 07:53 | |
| *** kingia has joined #openstack-infra | 07:53 | |
| *** penguinolog has joined #openstack-infra | 07:54 | |
| penguinolog | Good morning | 07:55 |
| *** asettle has quit IRC | 07:55 | |
| *** jlanoux has joined #openstack-infra | 07:55 | |
| *** kingia_ has quit IRC | 07:56 | |
| penguinolog | Guys, could anybody explain for me: is tox env docs enough for job template gate-xxx-docs (template python-jobs)? | 07:57 |
| penguinolog | changeset: https://review.openstack.org/#/c/292746/ | 08:00 |
| yolanda | AJaeger, question about https://review.openstack.org/#/c/289589/5/jenkins/jobs/projects.yaml . Why is the job definition duplicated here, one with job suffix nv and the other without? | 08:00 |
| *** Guest97519 has quit IRC | 08:00 | |
| AJaeger | penguinolog: we run "tox -e venv python setup.py build_sphinx" for the docs gate as explained in the PTI http://governance.openstack.org/reference/cti/python_cti.html | 08:01 |
| *** matrohon has joined #openstack-infra | 08:04 | |
| *** tesseract has joined #openstack-infra | 08:05 | |
| AJaeger | yolanda: let me check... | 08:05 |
| *** tesseract is now known as Guest91794 | 08:06 | |
| AJaeger | in this case: The -nv is not needed, let'S ask to remove it... | 08:06 |
| yolanda | ++ | 08:06 |
| AJaeger | In general: -nv for non-voting jobs, the other as voting | 08:06 |
| *** Qiming__ is now known as Qiming | 08:06 | |
| yolanda | yep, but i was seeing that dupe...normally just one definition needed right? | 08:06 |
| *** _nadya_ has joined #openstack-infra | 08:06 | |
| AJaeger | normally just one - unless they really need both - and they might. | 08:07 |
| AJaeger | Also, looking closer now: They don't use that job at all;( Ask them to add it to their check queue... | 08:07 |
| * AJaeger completely missed that - that was a immature +2 ;( | 08:07 | |
| yolanda | AJaeger, tsk tsk :) | 08:07 |
| yolanda | so you will add the comments yourself? | 08:08 |
| AJaeger | I can... | 08:08 |
| yolanda | ok | 08:08 |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: move python 2.7 checks to community https://review.openstack.org/292746 | 08:09 |
| AJaeger | done, yolanda | 08:10 |
| yolanda | cool | 08:10 |
| yolanda | AJaeger, i was thinking in approving https://review.openstack.org/#/c/292572 | 08:10 |
| yolanda | do you think we will be fine? i can monitor | 08:10 |
| *** flepied has quit IRC | 08:12 | |
| AJaeger | it should be fine, yes... | 08:12 |
| *** markvoelker has joined #openstack-infra | 08:12 | |
| yolanda | ok here it goes | 08:12 |
| AJaeger | 32 jobs only - so hope that goes through just fine... | 08:12 |
| openstackgerrit | Merged openstack-infra/project-config: zuul/layout: save TripleO CI resources https://review.openstack.org/292339 | 08:15 |
| AJaeger | yolanda: The followup will be more fun - and we should wait a day or two before we merge that | 08:15 |
| yolanda | i haven't looked. more jobs involved? | 08:15 |
| AJaeger | the remaining 414 devstack-trusty ones ... | 08:17 |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: move python 2.7 checks to community https://review.openstack.org/292746 | 08:17 |
| *** rossella_ has joined #openstack-infra | 08:17 | |
| AJaeger | yolanda: I already -1 any jobs that add new devstack-trusty (and also check for bare-trusty). | 08:18 |
| AJaeger | New trusty jobs all have to use ubuntu-trusty | 08:18 |
| yolanda | ++ i'll do the same then | 08:18 |
| *** salv-orl_ has quit IRC | 08:18 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting https://review.openstack.org/292687 | 08:18 |
| AJaeger | yolanda: https://review.openstack.org/#/c/292573 is the followup | 08:18 |
| yolanda | yay, it looks much more complex yes | 08:19 |
| *** fhubik has joined #openstack-infra | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Start devstack-trusty to ubuntu-trusty switch https://review.openstack.org/292572 | 08:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add Designate Role to OpenStack-Ansible https://review.openstack.org/292326 | 08:21 |
| openstackgerrit | Merged openstack-infra/project-config: Do not run gate jobs when doing docs-only patch https://review.openstack.org/292187 | 08:21 |
| openstackgerrit | Merged openstack-infra/project-config: ec2-api job: remove tempest from rally job https://review.openstack.org/292418 | 08:22 |
| *** arxcruz has joined #openstack-infra | 08:23 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable Ceilo plugin for Sahara dsvm jobs https://review.openstack.org/217751 | 08:23 |
| penguinolog | AJaeger, sorry, couild you check https://review.openstack.org/#/c/292746/ for consistency? (tox envs added: pylint, pep8, py27) | 08:24 |
| openstackgerrit | Merged openstack-infra/project-config: New project for TripleO UI https://review.openstack.org/291227 | 08:27 |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Add javascript jobs to fuel-ui project https://review.openstack.org/292760 | 08:31 |
| *** maishsk_ has joined #openstack-infra | 08:31 | |
| *** fawadkhaliq has joined #openstack-infra | 08:31 | |
| *** maishsk has quit IRC | 08:32 | |
| *** maishsk_ is now known as maishsk | 08:32 | |
| *** chlong has quit IRC | 08:32 | |
| *** vincentll has joined #openstack-infra | 08:33 | |
| *** [1]Thelo has joined #openstack-infra | 08:33 | |
| *** Thelo has quit IRC | 08:36 | |
| *** jlanoux has quit IRC | 08:36 | |
| *** [1]Thelo is now known as Thelo | 08:36 | |
| *** fhubik has quit IRC | 08:37 | |
| *** Hal has joined #openstack-infra | 08:38 | |
| *** Hal is now known as Guest6861 | 08:39 | |
| *** kzaitsev_mb_ has joined #openstack-infra | 08:40 | |
| *** markvoelker has quit IRC | 08:44 | |
| *** Guest91794 has quit IRC | 08:45 | |
| *** fhubik has joined #openstack-infra | 08:46 | |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: move python 2.7 checks to community https://review.openstack.org/292746 | 08:47 |
| tobiash | jhesketh: I just tried out this keepalive patch: https://review.openstack.org/#/c/238988/ | 08:47 |
| AJaeger | penguinolog: why do you +1 your own changes? | 08:47 |
| tobiash | jhesketh: It doesn't work for me because of a missing conversion of the config parameter. I added a -1 but the patch has already +2 from you so. | 08:48 |
| AJaeger | tobiash: other reviewers will see your -1 | 08:48 |
| *** kingia_ has joined #openstack-infra | 08:49 | |
| *** zeih has joined #openstack-infra | 08:50 | |
| *** kingia has quit IRC | 08:51 | |
| *** dizquierdo has joined #openstack-infra | 08:51 | |
| *** esikachev has joined #openstack-infra | 08:51 | |
| *** kingia has joined #openstack-infra | 08:53 | |
| *** kingia_ has quit IRC | 08:53 | |
| *** _degorenko|afk is now known as degorenko | 08:54 | |
| *** AJaeger has quit IRC | 08:54 | |
| tobiash | ok | 08:54 |
| *** AJaeger has joined #openstack-infra | 08:54 | |
| *** e0ne has joined #openstack-infra | 08:57 | |
| *** dizquierdo has quit IRC | 08:57 | |
| *** dizquierdo has joined #openstack-infra | 08:58 | |
| openstackgerrit | Evgeniy L proposed openstack-infra/project-config: Create bareon-allocator repository https://review.openstack.org/292765 | 08:58 |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting https://review.openstack.org/292766 | 08:58 |
| *** Guest91794 has joined #openstack-infra | 09:00 | |
| *** tzn has joined #openstack-infra | 09:01 | |
| *** zz_dimtruck is now known as dimtruck | 09:02 | |
| *** dtardivel has joined #openstack-infra | 09:03 | |
| *** HeOS has quit IRC | 09:03 | |
| *** HeOS has joined #openstack-infra | 09:04 | |
| *** flepied has joined #openstack-infra | 09:05 | |
| *** jordanP has joined #openstack-infra | 09:05 | |
| *** hashar has joined #openstack-infra | 09:05 | |
| *** HeOS has quit IRC | 09:05 | |
| *** HeOS has joined #openstack-infra | 09:06 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible https://review.openstack.org/292771 | 09:11 |
| *** sc68cal has quit IRC | 09:11 | |
| openstackgerrit | Alexey Ovchinnikov proposed openstack-infra/project-config: Add publishing capability to buildimage-lxd job https://review.openstack.org/290418 | 09:12 |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Add javascript jobs to fuel-ui project https://review.openstack.org/292760 | 09:12 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible https://review.openstack.org/292771 | 09:18 |
| *** zeih has quit IRC | 09:18 | |
| penguinolog | AJaeger, just follows standard rules inside team: until self +1, it's read as "forgot to set -W" | 09:19 |
| *** Daisy has quit IRC | 09:19 | |
| *** jistr has joined #openstack-infra | 09:20 | |
| AJaeger | penguinolog: that's not a standard OpenStack rule... Your +1 does not harm, it's just unusual in most (nearly all?) projects. | 09:20 |
| *** salv-orlando has joined #openstack-infra | 09:21 | |
| penguinolog | AJaeger, fixed: set to 0 | 09:21 |
| penguinolog | could anybody else review https://review.openstack.org/#/c/292746/5 ? | 09:22 |
| odyssey4me | heh, agreed AJaeger - never heard of that convention myself | 09:22 |
| *** hashar has quit IRC | 09:22 | |
| odyssey4me | if you're still working on it, mark it -w or at least set the subject to some form of 'WIP' | 09:23 |
| *** andymaier has joined #openstack-infra | 09:23 | |
| *** Guest6861 has quit IRC | 09:24 | |
| *** hashar has joined #openstack-infra | 09:25 | |
| *** jlanoux has joined #openstack-infra | 09:25 | |
| *** Hal has joined #openstack-infra | 09:27 | |
| *** Hal is now known as Guest89920 | 09:28 | |
| *** denisra has quit IRC | 09:28 | |
| *** denisra has joined #openstack-infra | 09:29 | |
| *** salv-orlando has quit IRC | 09:29 | |
| *** salv-orlando has joined #openstack-infra | 09:30 | |
| *** rcernin has joined #openstack-infra | 09:31 | |
| *** tzn has quit IRC | 09:33 | |
| *** lucas-dinner is now known as lucasagomes | 09:33 | |
| *** kushal has quit IRC | 09:35 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Pre-release stable/mitaka ACLs https://review.openstack.org/292781 | 09:36 |
| *** zhurong_ has quit IRC | 09:37 | |
| *** maishsk has quit IRC | 09:40 | |
| *** watersoul has joined #openstack-infra | 09:44 | |
| *** watersoul_ has quit IRC | 09:47 | |
| *** derekh has joined #openstack-infra | 09:47 | |
| *** hichihara has quit IRC | 09:52 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 09:53 |
| *** mrmartin has joined #openstack-infra | 09:53 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting https://review.openstack.org/292687 | 09:58 |
| *** Qiming has quit IRC | 09:59 | |
| *** claudiub has joined #openstack-infra | 09:59 | |
| *** sdague has joined #openstack-infra | 10:01 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TEST: overcloud SSL with netiso https://review.openstack.org/292787 | 10:01 |
| *** ociuhandu has quit IRC | 10:05 | |
| *** gildub has quit IRC | 10:07 | |
| *** rossella_ has quit IRC | 10:07 | |
| *** rossella_s has joined #openstack-infra | 10:08 | |
| *** sc68cal has joined #openstack-infra | 10:08 | |
| *** e0ne has quit IRC | 10:08 | |
| *** e0ne has joined #openstack-infra | 10:08 | |
| *** jcoufal has joined #openstack-infra | 10:10 | |
| *** Guest89920 has quit IRC | 10:11 | |
| *** pcrews_ has quit IRC | 10:11 | |
| *** tpsilva has joined #openstack-infra | 10:11 | |
| *** sc68cal has quit IRC | 10:13 | |
| tobiash | Hi, I've setup a new zuul (current master) + jenkins (1.642.2) environment and somehow no build urls get passed back to zuul. | 10:14 |
| *** pcrews_ has joined #openstack-infra | 10:14 | |
| tobiash | Zuul seems to get update messages when a build starts, but they don't contain the build url | 10:15 |
| *** sambetts|afk is now known as sambetts | 10:15 | |
| tobiash | Is there something I have to configure in jenkins (it knows it's base url)? | 10:16 |
| openstackgerrit | yolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce https://review.openstack.org/292792 | 10:16 |
| yolanda | AJaeger, fix for devstack-gate and reproduce ^ | 10:17 |
| *** mrmartin has quit IRC | 10:17 | |
| eliqiao | hi AJaeger | 10:18 |
| eliqiao | AJaeger: I have a question about https://review.openstack.org/#/c/292687/3/zuul/layout.yaml | 10:18 |
| *** mkj has quit IRC | 10:19 | |
| *** electrofelix has joined #openstack-infra | 10:21 | |
| *** AJaeger has quit IRC | 10:21 | |
| *** notnownikki has joined #openstack-infra | 10:23 | |
| jhesketh | tobiash: did casting it to an int work for you? What did you have in your config file (eg was it a string) | 10:24 |
| *** oanson has joined #openstack-infra | 10:24 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 10:26 |
| *** rguillebert has joined #openstack-infra | 10:26 | |
| tobiash | jhesketh: yepp, in the config file I had "keepalive=60" | 10:27 |
| tobiash | without the casting paramiko threw errors | 10:27 |
| tobiash | with casting it works for me | 10:27 |
| jhesketh | hmm, I wonder why gerrit's port works... maybe nobody has tried actually setting it | 10:28 |
| *** flwang1 has joined #openstack-infra | 10:28 | |
| yolanda | jhesketh, care reviewing https://review.openstack.org/292521 ? has a +2 from AJaeger, is to enable a nv job in bifrost | 10:29 |
| igorbelikov | hey folks, is zuul stuck? no changes in queues at http://status.openstack.org/zuul/ and the graphs seem flatlining | 10:29 |
| jhesketh | tobiash: oh, nevermind, it has an int | 10:29 |
| tobiash | jhesketh: zuul itself seemed to work, I saw the errors in the log (something like operator + not supported for float and string) | 10:29 |
| flwang1 | hi guys, how can i create a new group on gerrit for a project? thanks | 10:29 |
| jhesketh | tobiash: interesting that there's a + operator on a port number... got a copy of that log per hcnace? | 10:30 |
| tobiash | jhesketh: have to look | 10:31 |
| *** jlanoux_ has joined #openstack-infra | 10:32 | |
| *** jlanoux has quit IRC | 10:32 | |
| tobiash | jhesketh: http://paste.openstack.org/show/490466/ | 10:33 |
| tobiash | with casting this error goes away | 10:33 |
| jhesketh | tobiash: oh right, for the keepalive | 10:34 |
| jhesketh | tobiash: yeah, looks correct. Nice catch :-) | 10:34 |
| tobiash | :) | 10:34 |
| jhesketh | yolanda: lgtm | 10:36 |
| yolanda | jhesketh, thx | 10:36 |
| *** vgridnev has joined #openstack-infra | 10:36 | |
| jhesketh | yolanda: if you have time for https://review.openstack.org/#/c/289125/ that'd be appreciated | 10:37 |
| *** esikachev has quit IRC | 10:37 | |
| yolanda | sure | 10:37 |
| *** sfinucan has joined #openstack-infra | 10:37 | |
| *** shardy has joined #openstack-infra | 10:39 | |
| jhesketh | thanks :-) | 10:40 |
| *** dimtruck is now known as zz_dimtruck | 10:40 | |
| *** zz_dimtruck is now known as dimtruck | 10:42 | |
| *** esikachev has joined #openstack-infra | 10:42 | |
| *** markvoelker has joined #openstack-infra | 10:42 | |
| *** x00350071 has quit IRC | 10:42 | |
| igorbelikov | yolanda: zuul seems stuck for the past 2 hours, no new changes in queues, is this a known situation? | 10:43 |
| yolanda | igorbelikov, i was not aware | 10:43 |
| yolanda | checking | 10:43 |
| *** x00350071 has joined #openstack-infra | 10:43 | |
| *** esikachev has quit IRC | 10:44 | |
| *** tzn has joined #openstack-infra | 10:44 | |
| igorbelikov | yolanda: thanks! | 10:44 |
| *** esikachev has joined #openstack-infra | 10:44 | |
| yolanda | jhesketh, still around? i see lots of | 10:46 |
| yolanda | 2016-03-15 10:45:57,094 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent ref-replicated None> | 10:46 |
| yolanda | 2016-03-15 10:45:57,097 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Can not get account information. | 10:46 |
| yolanda | in zuul logs | 10:46 |
| yolanda | it seems to get trigger events, but slowly | 10:46 |
| jhesketh | yolanda: those errors have been there since the gerrit upgrade (afaik) | 10:46 |
| yolanda | mm, red herring then | 10:47 |
| jhesketh | the no changes thingy is less cool | 10:47 |
| yolanda | new changes are not queued | 10:47 |
| jhesketh | yolanda: maybe check an older log just to confirm | 10:47 |
| *** andymaier has quit IRC | 10:47 | |
| jhesketh | yolanda: did it reload recently? it might be recalcaulting dependencies etc | 10:47 |
| jhesketh | that usually holds it up for a little (although probably not hours) | 10:47 |
| yolanda | i'm seeing several "Done adding trigger event" on latest entries | 10:47 |
| jhesketh | then again, I had a 500 error from gerrit, so maybe that's slowing things down | 10:48 |
| yolanda | mm, i tried telnet to port 4730, and STATUS command is not responding | 10:50 |
| yolanda | jhesketh, do you think there can be another alternative, apart from zuul restart? | 10:54 |
| jhesketh | gearman isn't responding? :-s | 10:54 |
| yolanda | i can telnet but doesn't respond to any command | 10:55 |
| jhesketh | yolanda: did you do 'status' lower case? | 10:55 |
| * jhesketh isn't sure if it's case sensitive | 10:55 | |
| yolanda | ouch, uppercase | 10:55 |
| yolanda | status works | 10:56 |
| yolanda | :( | 10:56 |
| jhesketh | right, the load and memory look okay on the server, so I'm not sure we need to restart zuul just yet | 10:56 |
| yolanda | log is strange, not collecting any new change, not interacting with jenkins... | 10:56 |
| igorbelikov | yolanda jhesketh looks like about the same time our fuel ci stopped posting comments on gerrit via gerrit-trigger. There’s a queue of unprocessed gerrit events in jenkins logs, so maybe this is on gerrit side? | 10:57 |
| *** AJaeger has joined #openstack-infra | 10:57 | |
| yolanda | i cannot see errors on gerrit log | 10:58 |
| *** gildub has joined #openstack-infra | 10:59 | |
| yolanda | and i did a recheck of my change, and zuul picked that properly | 10:59 |
| yolanda | 2016-03-15 10:45:57,094 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent ref-replicated None> | 10:59 |
| yolanda | 2016-03-15 10:45:57,097 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Can not get account information. | 10:59 |
| yolanda | mm, not that | 10:59 |
| yolanda | 2016-03-15 10:58:37,602 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent comment-added openstack-infra/devstack-gate master 292792,1> | 10:59 |
| yolanda | so it can get events from gerrit but is not adding them to the queue | 10:59 |
| jhesketh | if zuul is stuck posting results back it may not process the event queue | 11:00 |
| *** Qiming has joined #openstack-infra | 11:00 | |
| yolanda | only entries in zuul right now, are the adding trigger event ones | 11:00 |
| igorbelikov | http://paste.openstack.org/show/490471/ - I see a lot of this in our logs. Our jenkins picks up changes fine, but has problems posting back results, and this is via gerrit-trigger, so zuul is not involved in the process | 11:01 |
| *** maishsk has joined #openstack-infra | 11:02 | |
| jhesketh | yolanda: do you see any "Processing result event" in the log? | 11:02 |
| *** mizar_ has joined #openstack-infra | 11:03 | |
| *** mizar_ is now known as flongo | 11:03 | |
| *** fabio_ has joined #openstack-infra | 11:03 | |
| jhesketh | yolanda: and is there a "Processing trigger event" afterwards | 11:03 |
| *** fabio_ has quit IRC | 11:03 | |
| jhesketh | (ie has it gotten stuck on the result) | 11:03 |
| yolanda | jhesketh, no Processing result since 08:34, 2 hours and a half before | 11:04 |
| *** fverbs has joined #openstack-infra | 11:04 | |
| jhesketh | yolanda: but any trigger event afterwards? | 11:04 |
| yolanda | jhesketh, yes, it continues posting logs about Adding trigger event | 11:05 |
| yolanda | the Adding trigger/Done adding trigger pair | 11:05 |
| jhesketh | yolanda: that's in another thread... are there any "Processing trigger event" ? | 11:05 |
| yolanda | no | 11:06 |
| *** AJaeger_ has joined #openstack-infra | 11:06 | |
| odyssey4me | yolanda odd, it seems that https://review.openstack.org/292326 has merged and executed, but http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate is empty | 11:06 |
| *** kushal has joined #openstack-infra | 11:07 | |
| jhesketh | yolanda: okay so it's stuck posting results back | 11:07 |
| *** AJaeger__ has joined #openstack-infra | 11:08 | |
| yolanda | jhesketh, looks like. In status page there are several jobs with all entries in SUCCESS, but they are still there and not reported | 11:08 |
| *** x00350071 has quit IRC | 11:08 | |
| yolanda | what should be best way to unlock it? | 11:08 |
| jhesketh | yolanda: well there are 317 results waiting to be posted back | 11:08 |
| jhesketh | yolanda: not sure... (without losing the work anyway) | 11:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible https://review.openstack.org/292771 | 11:09 |
| yolanda | yep, a restart will do it, but it will cause that people need to recheck their jobs | 11:09 |
| *** AJaeger has quit IRC | 11:10 | |
| jhesketh | yolanda: I'm also not sure if gerrit is having trouble | 11:10 |
| yolanda | yes, it can be gerrit... but i don't see errors on logs, and queue is empty | 11:10 |
| yolanda | how can verify it? | 11:10 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 11:10 |
| yolanda | also a gerrit restart is less harmful than zuul | 11:11 |
| jhesketh | since igorbelikov's CI stopped processing events and it hasn't repliacted odyssey4me's changes | 11:11 |
| *** AJaeger_ has quit IRC | 11:11 | |
| jhesketh | yolanda: I'm actually not sure what effect it'll have on a zuul that's stuck posting back results | 11:11 |
| *** moravec has joined #openstack-infra | 11:11 | |
| jhesketh | but it might be safer | 11:11 |
| yolanda | ok let's try first a gerrit restart... | 11:12 |
| *** dims has joined #openstack-infra | 11:12 | |
| *** rhallisey has joined #openstack-infra | 11:12 | |
| yolanda | #status notice Gerrit is going to be restarted | 11:12 |
| openstackstatus | yolanda: sending notice | 11:12 |
| * jhesketh doesn't like restarting things randomly as guesses but doesn't have a better suggestion | 11:12 | |
| yolanda | yes, at this point we don't have better options | 11:12 |
| yolanda | mm, it unlocked zuul actually | 11:13 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted | 11:13 | |
| sdague | well, I don't think gerrit is having any more 500 errors than normal | 11:14 |
| yolanda | i start to see zuul submiting changes | 11:14 |
| jhesketh | yolanda: right, that particular result may have been lost and it should continue through the result queue | 11:14 |
| sdague | the even queue length is still super high | 11:14 |
| openstackstatus | yolanda: finished sending notice | 11:15 |
| *** wei-wang has quit IRC | 11:15 | |
| *** markvoelker has quit IRC | 11:15 | |
| yolanda | i sent a recheck and i still don't see in queue | 11:15 |
| sdague | well with the event queue over 3000, that's not totally surprising, right? | 11:16 |
| yolanda | zuul log looks much better now | 11:16 |
| yolanda | it's weird that gerrit fails this way, the logs were not really showing a problem | 11:16 |
| yolanda | apart from the normal gerrit problems | 11:17 |
| jhesketh | yolanda: the result queue doesn't appear to be going down | 11:17 |
| jhesketh | but all the check jobs have disappeared | 11:17 |
| jhesketh | (that's possible because I think they were all done since the other changes are waiting in the event queue) | 11:18 |
| yolanda | i see some entries for merged jobs in zuul, and posting events as well | 11:18 |
| jhesketh | yolanda: oh, there it goes | 11:18 |
| yolanda | yes, check is there now | 11:18 |
| jhesketh | and the event queue is being processed | 11:18 |
| yolanda | cool | 11:19 |
| jhesketh | looks like nodepool is responding to the load appropriately | 11:20 |
| *** dims has quit IRC | 11:20 | |
| yolanda | mm, there are lots of old entries in gerrit.log about loosing database connection | 11:21 |
| *** dims has joined #openstack-infra | 11:21 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:21 | |
| *** dims has quit IRC | 11:21 | |
| *** dims has joined #openstack-infra | 11:21 | |
| AJaeger__ | jhesketh: https://review.openstack.org/#/c/289125/6/jenkins/jobs/xstatic.yaml does not install any distro packages | 11:22 |
| AJaeger__ | Is that correct? | 11:22 |
| jhesketh | AJaeger__: it was copied from the tarball job... should it install any? | 11:23 |
| AJaeger__ | jhesketh: ah, you just do tarballs - should be fine... | 11:23 |
| jhesketh | :-) | 11:25 |
| AJaeger__ | I think gerrit did not record any events from those jobs that were finished ;(, so I rechecked a few ... | 11:26 |
| yolanda | AJaeger yes, seems that several job postings were lost | 11:26 |
| jhesketh | oh, that sucks :-s | 11:27 |
| *** mrunge has quit IRC | 11:28 | |
| AJaeger__ | yolanda: what about a notice to ask for rechecks? | 11:28 |
| yolanda | makes sense, i'll send | 11:28 |
| *** ldnunes has joined #openstack-infra | 11:29 | |
| AJaeger__ | yolanda: give a timeframe if possible | 11:29 |
| *** ihrachys has quit IRC | 11:31 | |
| *** ihrachys has joined #openstack-infra | 11:31 | |
| yolanda | #status notice Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 08:30 UTC to 10:30 UTC approximately. Please recheck any affected jobs by this problem. | 11:32 |
| openstackstatus | yolanda: sending notice | 11:32 |
| *** mrmartin has joined #openstack-infra | 11:32 | |
| yolanda | mm, i think i messed with timezone... is 11:30 utc :( | 11:32 |
| *** tzn has quit IRC | 11:32 | |
| AJaeger__ | yolanda: approximately ;) | 11:32 |
| flongo | hi all, I do not know if my review is affected by the problem: https://review.openstack.org/#/c/292801/ | 11:33 |
| -openstackstatus- NOTICE: Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 08:30 UTC to 10:30 UTC approximately. Please recheck any affected jobs by this problem. | 11:33 | |
| yolanda | #status notice Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 09:30 UTC to 11:30 UTC approximately. Please recheck any affected jobs by this problem. | 11:33 |
| flongo | how long does it take to jenkins before starting to test a review that has been approved? | 11:33 |
| *** tzn has joined #openstack-infra | 11:33 | |
| yolanda | flongo, it depends on the load. I suggest you check http://status.openstack.org/zuul. If you don't see your change there, and you don't have any results posted on gerrit, please recheck | 11:34 |
| openstackstatus | yolanda: finished sending notice | 11:34 |
| openstackstatus | yolanda: sending notice | 11:34 |
| -openstackstatus- NOTICE: Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 09:30 UTC to 11:30 UTC approximately. Please recheck any affected jobs by this problem. | 11:35 | |
| *** ihrachys has quit IRC | 11:36 | |
| AJaeger__ | flongo, your change is in the check queue. | 11:36 |
| AJaeger__ | And once it passes that one it goes into gate queue... | 11:36 |
| *** kushal has quit IRC | 11:37 | |
| openstackstatus | yolanda: finished sending notice | 11:37 |
| flongo | yolanda: yes I saw it in the queue, thanks | 11:37 |
| flongo | Ajaeger__ thanks! | 11:38 |
| *** weshay has joined #openstack-infra | 11:39 | |
| *** mrunge has joined #openstack-infra | 11:40 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Update Email Configuration file https://review.openstack.org/292605 | 11:41 |
| *** thorst_ has joined #openstack-infra | 11:41 | |
| *** dizquierdo_afk is now known as dizquierdo | 11:42 | |
| *** jed56 has quit IRC | 11:43 | |
| *** rfolco has joined #openstack-infra | 11:45 | |
| *** Hal has joined #openstack-infra | 11:45 | |
| *** Hal is now known as Guest27719 | 11:46 | |
| *** nmagnezi has quit IRC | 11:46 | |
| *** gildub has quit IRC | 11:48 | |
| *** erlon has joined #openstack-infra | 11:49 | |
| *** nmagnezi has joined #openstack-infra | 11:49 | |
| *** jlanoux has joined #openstack-infra | 11:53 | |
| *** _amrith_ is now known as amrith | 11:54 | |
| *** salv-orl_ has joined #openstack-infra | 11:54 | |
| *** jlanoux_ has quit IRC | 11:56 | |
| *** salv-orlando has quit IRC | 11:57 | |
| *** kgiusti has joined #openstack-infra | 11:58 | |
| *** trown has joined #openstack-infra | 12:00 | |
| *** HeOS has quit IRC | 12:00 | |
| *** HeOS has joined #openstack-infra | 12:01 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 12:03 |
| *** mrmartin has quit IRC | 12:04 | |
| *** ociuhandu has joined #openstack-infra | 12:05 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Update Zuul Layout for Puppet OpenStack https://review.openstack.org/292555 | 12:05 |
| *** zhurong_ has joined #openstack-infra | 12:07 | |
| *** ihrachys has joined #openstack-infra | 12:07 | |
| *** dtardivel has quit IRC | 12:08 | |
| *** salv-orl_ has quit IRC | 12:08 | |
| *** jaosorior has quit IRC | 12:08 | |
| *** aysyd has joined #openstack-infra | 12:08 | |
| *** jaosorior has joined #openstack-infra | 12:09 | |
| *** jaosorior has quit IRC | 12:14 | |
| *** shardy has quit IRC | 12:16 | |
| *** thiagop has joined #openstack-infra | 12:16 | |
| thiagop | Hello folks, I'm having some trouble setting up a CI server with the puppet modules wrt a RSA key being mistaken as ECDSA. Does anybody know how to solve this? | 12:18 |
| *** zhurong_ has quit IRC | 12:18 | |
| thiagop | quick google showed how to solve this direclty on paramiko, but I don't think it should need to change code to do that... | 12:19 |
| rcarrillocruz | thiagop: when you say set up CI server, what do you mean? | 12:19 |
| rcarrillocruz | is it a gerrit | 12:19 |
| rcarrillocruz | a zuul | 12:20 |
| rcarrillocruz | ... | 12:20 |
| rcarrillocruz | ? | 12:20 |
| rcarrillocruz | which puppet module are you seeing that failure | 12:20 |
| thiagop | rcarrillocruz: yes, all of that. Using the single_node_ci.pp | 12:20 |
| *** bdemers has quit IRC | 12:20 | |
| rcarrillocruz | so you mean puppet-openstackci | 12:20 |
| thiagop | rcarrillocruz: yep | 12:20 |
| *** _david_ has joined #openstack-infra | 12:20 | |
| rcarrillocruz | paste the site.pp you are using | 12:21 |
| rcarrillocruz | the command | 12:21 |
| rcarrillocruz | and the full error | 12:21 |
| rcarrillocruz | on paste.openstack.org | 12:21 |
| thiagop | rcarrillocruz: in fact, the error is on zuul.log after the installation completes | 12:21 |
| *** sfinucan has quit IRC | 12:22 | |
| thiagop | rcarrillocruz: http://paste.openstack.org/show/490495/ | 12:22 |
| _david_ | fungi, jeblair, mordred We need a feedback if AccountPatchReview thing is important for you to stay, or can be removed | 12:22 |
| *** sfinucan has joined #openstack-infra | 12:22 | |
| _david_ | ^^^ https://groups.google.com/d/topic/repo-discuss/KhXKMKTJNMs/discussion | 12:22 |
| *** ayoung has joined #openstack-infra | 12:23 | |
| rcarrillocruz | do you have this zuul pointing at your own gerrit | 12:25 |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box' https://review.openstack.org/286137 | 12:25 |
| rcarrillocruz | or to infra gerrit , i.e. review.openstack.org | 12:25 |
| rcarrillocruz | ? | 12:25 |
| *** dims has quit IRC | 12:25 | |
| AJaeger__ | fungi, I updated https://etherpad.openstack.org/p/bindep with some findings about database setup including some questions. Could you extend/comment, please? | 12:25 |
| *** dims has joined #openstack-infra | 12:26 | |
| thiagop | rcarrillocruz: no, it's pointed to openstack's. And the key is already on our user's account (I tested the key reading the stream with SSH) | 12:26 |
| rcarrillocruz | are you sure your zuul.conf points to the correct ssh private file? | 12:28 |
| *** piet has joined #openstack-infra | 12:29 | |
| *** Guest27719 has quit IRC | 12:30 | |
| thiagop | rcarrillocruz: yeah, the file is generated using the key we paste on common.yaml and I already tested the key for format issues using it on SSH | 12:30 |
| thiagop | rcarrillocruz: the key path on zuul.conf is correct too | 12:30 |
| rcarrillocruz | and if you do | 12:32 |
| rcarrillocruz | sudo su - zuul | 12:32 |
| rcarrillocruz | you can cat just fine the sshkey you have on zuul.conf | 12:32 |
| rcarrillocruz | ? | 12:32 |
| *** baoli has joined #openstack-infra | 12:32 | |
| *** edmondsw has joined #openstack-infra | 12:32 | |
| rcarrillocruz | i'm asking all these questions cos that error is a pretty generic one | 12:33 |
| rcarrillocruz | and i've encountered it because of missing param | 12:33 |
| rcarrillocruz | or lack of perms on zuul user to access it | 12:33 |
| *** baoli_ has joined #openstack-infra | 12:33 | |
| rcarrillocruz | i never encountered an issue about ecdsa | 12:33 |
| *** armax has quit IRC | 12:33 | |
| thiagop | rcarrillocruz: cat works fine | 12:33 |
| *** hichihara has joined #openstack-infra | 12:35 | |
| thiagop | rcarrillocruz: ssh -i using key and from zuul user works fine too | 12:36 |
| *** jaosorior has joined #openstack-infra | 12:36 | |
| *** baoli has quit IRC | 12:37 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** trown has quit IRC | 12:43 | |
| *** shardy has joined #openstack-infra | 12:44 | |
| *** sulo has quit IRC | 12:44 | |
| *** lucasagomes is now known as lucas-hungry | 12:45 | |
| *** sulo has joined #openstack-infra | 12:45 | |
| *** sulo has quit IRC | 12:46 | |
| *** asettle has joined #openstack-infra | 12:46 | |
| *** sulo has joined #openstack-infra | 12:46 | |
| *** shardy has quit IRC | 12:47 | |
| *** shardy has joined #openstack-infra | 12:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TEST: overcloud SSL with netiso https://review.openstack.org/292787 | 12:49 |
| *** gordc has joined #openstack-infra | 12:50 | |
| *** amrith is now known as _amrith_ | 12:50 | |
| AJaeger__ | yolanda: 292521 is not entering teh gate queue - is zuul buggy as well and refusing to add anythign to gate? | 12:51 |
| *** asettle has quit IRC | 12:51 | |
| yolanda | going to check logs, yes... is weird that it doesn't get added | 12:51 |
| *** AJaeger__ is now known as AJaeger | 12:52 | |
| *** dprince has joined #openstack-infra | 12:52 | |
| igorbelikov | yolanda: same for 291762 and 291724 | 12:52 |
| *** trown has joined #openstack-infra | 12:53 | |
| *** markvoelker has joined #openstack-infra | 12:53 | |
| yolanda | mm | 12:53 |
| yolanda | 2016-03-15 12:47:33,073 DEBUG zuul.DependentPipelineManager: Considering adding change <Change 0x7f50847e78d0 292521,1> | 12:53 |
| yolanda | 2016-03-15 12:47:33,073 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f50847e78d0 292521,1> can not merge, ignoring | 12:53 |
| yolanda | 2016-03-15 12:47:33,073 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f50847e78d0 292521,1> is not ready to be enqueued, ignoring | 12:53 |
| AJaeger | yolanda: and repo openstack-ansible-os_designate seems to exist but is not setup completely ;( | 12:53 |
| *** ddieterly has joined #openstack-infra | 12:54 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Start adding test for bifrost dhcp https://review.openstack.org/292521 | 12:54 |
| yolanda | i rebased, let's see | 12:54 |
| AJaeger | yolanda: other jobs are not entering either like 92074 | 12:54 |
| AJaeger | I meant: 292074 | 12:54 |
| AJaeger | Oh, 286756,6 just entered gate! | 12:55 |
| AJaeger | sahara-elements | 12:55 |
| yolanda | i don't see errors for 292074 | 12:55 |
| yolanda | igorbelikov, i saw the same "not ready to be enqueued" error for 291762 | 12:56 |
| openstackgerrit | yolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce https://review.openstack.org/292792 | 12:58 |
| *** ddieterly is now known as ddieterly[away] | 13:01 | |
| *** ddieterly[away] is now known as ddieterly | 13:01 | |
| *** eharney has joined #openstack-infra | 13:01 | |
| *** trown is now known as trown|ooobootcam | 13:02 | |
| *** armax has joined #openstack-infra | 13:05 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Pre-release stable/mitaka ACLs https://review.openstack.org/292781 | 13:05 |
| *** dizquierdo has quit IRC | 13:05 | |
| *** yamamoto has quit IRC | 13:05 | |
| *** sdake has joined #openstack-infra | 13:08 | |
| *** maishsk has quit IRC | 13:08 | |
| *** maishsk has joined #openstack-infra | 13:08 | |
| SamYaple | lol mordred "First and foremost - I am the only ATC in the project currently." | 13:09 |
| SamYaple | thats a pretty solid reason | 13:09 |
| *** jlanoux has quit IRC | 13:10 | |
| pc_m | Hi! I added coverage testing to a repo (similar to what Rally project, where it compares coverage of change set to latest). Getting failures frequently, which are due to "build timed out". Is there anything that can be done? | 13:10 |
| dhellmann | we have a stuck review that we can't make merge: https://review.openstack.org/#/c/292539/ We've tried rechecking and adding another workflow. Is there anything else we can do to get it to merge? | 13:10 |
| *** jlanoux has joined #openstack-infra | 13:10 | |
| *** ddieterly has quit IRC | 13:11 | |
| dhellmann | AJaeger, fungi : do either of you have any suggestions? ^^ | 13:11 |
| sdague | dhellmann: there was a gerrit / zuul outage overnight | 13:11 |
| AJaeger | dhellmann: zuul/gerrit is somehow ill ;( yolanda has tried to heal it but that was only a band aid ;( | 13:11 |
| dhellmann | sdague : yeah, we've been trying to land it since then | 13:12 |
| *** yamahata has joined #openstack-infra | 13:12 | |
| dhellmann | ah, ok, so ongoing issues | 13:12 |
| AJaeger | dhellmann: I fear we need the full fire brigade to wake up ;( | 13:12 |
| *** ddieterly has joined #openstack-infra | 13:12 | |
| dhellmann | ok, we've done the release for that patch so it can wait, thanks AJaeger and yolanda | 13:13 |
| AJaeger | dhellmann: do you have some time for a chat? Then I join you in #openstack-releases | 13:13 |
| sdague | dhellmann: are you 100% it's cleanly rebased on master? I've found that there seems to be times where things wedge for this reason and it doesn't tell you | 13:13 |
| dhellmann | sdague : I'm not, so we could try that, too | 13:14 |
| sdague | I would try that as a work around, hit the rebase button, and see if you get different results | 13:14 |
| *** akshai has joined #openstack-infra | 13:14 | |
| *** amitgandhinz has joined #openstack-infra | 13:15 | |
| AJaeger | sdague: I really doubt it based on what I've seen. But let's try... | 13:16 |
| yolanda | not sure what's going on now | 13:16 |
| *** salv-orlando has joined #openstack-infra | 13:16 | |
| dims | sdague : AJaeger : trying that trick now | 13:16 |
| *** ddieterly has quit IRC | 13:17 | |
| SamYaple | hello, can someone help me with this skip-if that just merged https://review.openstack.org/#/c/292187/ | 13:18 |
| yolanda | same error again, not arriving to gate | 13:18 |
| SamYaple | it does not appear to be working https://review.openstack.org/#/c/292869/ | 13:18 |
| *** ig0r__ has joined #openstack-infra | 13:19 | |
| SamYaple | that patch is running all gates and it should not based on that skip-if above | 13:19 |
| igorbelikov | yolanda: I have at least 7 changes with the same issue, so yeah, something is definitely wrong:( | 13:20 |
| yolanda | gate is just empty | 13:20 |
| yolanda | so something is quite wrong yes | 13:20 |
| *** piet has quit IRC | 13:20 | |
| *** tonytan4ever has joined #openstack-infra | 13:21 | |
| *** armax has quit IRC | 13:21 | |
| *** asettle has joined #openstack-infra | 13:21 | |
| *** ig0r_ has quit IRC | 13:22 | |
| *** greghaynes has quit IRC | 13:22 | |
| *** jlanoux has quit IRC | 13:25 | |
| *** chlong has joined #openstack-infra | 13:25 | |
| *** asettle has quit IRC | 13:25 | |
| yolanda | any other infra-root can help with the issue? right now i cannot see more options than a zuul restart | 13:26 |
| *** ayoung has quit IRC | 13:26 | |
| *** jlanoux has joined #openstack-infra | 13:26 | |
| yolanda | i just keep seeing that "not ready to be enqueued, ignoring" error | 13:27 |
| dims | yolanda: that's why the gate queue is empty. right? | 13:29 |
| *** dimtruck is now known as zz_dimtruck | 13:29 | |
| fungi | yolanda: jhesketh: it's possible if gerrit lost its network socket to trove for some reason that it could have been blocking zuul on a report call from that timeframe | 13:29 |
| yolanda | yes, all changes going to gate seem to hit that problem | 13:29 |
| *** mriedem_away is now known as mriedem | 13:30 | |
| yolanda | fungi but what seems strange is that check queue looks fine now, and gate fails | 13:31 |
| fungi | thiagop: if memory serves, openssh will try several configured key types, so if it doesn't recognize the rsa key as valid it'll look for a dsa one, then an ecdsa one, then it'll emit a misleading error about not finding a valid ecdsa key even though the underlying cause is a corrupt/malformed rsa key | 13:31 |
| yolanda | if there was some block, zuul won't be operating... | 13:31 |
| yolanda | fungi, also.. zuul-server -t should be validating syntax properly? is that reilable? i ran it, to ensure that layout.yaml is valid, and i got a failure | 13:32 |
| *** rfolco has quit IRC | 13:33 | |
| *** larsbutler has quit IRC | 13:33 | |
| *** rfolco has joined #openstack-infra | 13:33 | |
| thiagop | fungi: do you think "ssh" is fixing the key somehow? I thought if I could connect using the same key, that should work with zuul too.. | 13:33 |
| *** piet has joined #openstack-infra | 13:34 | |
| *** piet has quit IRC | 13:34 | |
| *** infra-red has joined #openstack-infra | 13:35 | |
| *** greghaynes has joined #openstack-infra | 13:38 | |
| *** sdake has quit IRC | 13:38 | |
| *** Shrews has joined #openstack-infra | 13:38 | |
| yolanda | fungi, any ideas apart from trying to restart zuul? | 13:39 |
| *** pradk has joined #openstack-infra | 13:40 | |
| *** _ari_ has joined #openstack-infra | 13:40 | |
| *** ayoung has joined #openstack-infra | 13:40 | |
| *** akshai has quit IRC | 13:42 | |
| AJaeger | fungi, yolanda : The repos that have been created today like tripleo-ui and openstack-ansible-os_designate are not setup on git.openstack.org ;( | 13:43 |
| *** akshai has joined #openstack-infra | 13:44 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:45 | |
| *** akshai has quit IRC | 13:45 | |
| yolanda | AJaeger, are they created on gerrit properly? | 13:45 |
| *** akshai has joined #openstack-infra | 13:45 | |
| yolanda | last entry in logs for manage_projects is from 10 Mar, that's not good | 13:46 |
| fungi | _david_: i'm not aware of any of our existing tooling querying the account_patch_reviews table in mysql if that's what you're asking. perhaps jeblair is aware of something, but http://codesearch.openstack.org/ isn't turning up any direct use | 13:47 |
| *** mgrosz has joined #openstack-infra | 13:47 | |
| *** lucas-hungry is now known as lucasagomes | 13:48 | |
| *** nwkarsten has joined #openstack-infra | 13:48 | |
| *** nwkarsten has quit IRC | 13:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Only publish API guides from master https://review.openstack.org/292888 | 13:48 |
| yolanda | fungi, AJaeger, also puppetdb doesn't seem to be good for review.openstack.org | 13:48 |
| _david_ | fungi, thanks. | 13:48 |
| AJaeger | yolanda: I didn't check further | 13:48 |
| *** automagically_ has joined #openstack-infra | 13:48 | |
| _david_ | fungi, This is something you can use on the change screen: check box if a path was reviewed already or not | 13:48 |
| *** nwkarsten has joined #openstack-infra | 13:48 | |
| _david_ | (and diff side by side screen) | 13:48 |
| *** mgrosz has quit IRC | 13:48 | |
| *** automagically_ has left #openstack-infra | 13:49 | |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| kzaitsev_mb_ | hi, I think, that our patches in murano are not scheduled for gate checs and are not merged as result — is this a known issue? | 13:49 |
| *** kzaitsev_mb_ is now known as kzaitsev_mb | 13:49 | |
| *** yamamoto_ has joined #openstack-infra | 13:49 | |
| AJaeger | kzaitsev_mb: yes, a known issue. We have a generic problem with our CI | 13:50 |
| kzaitsev_mb | k, thanks not bugging you then =) | 13:50 |
| *** Eyal_ has joined #openstack-infra | 13:50 | |
| yolanda | doesn't look good | 13:50 |
| yolanda | fatal: [87741fa9-92cb-4d1e-8195-6841e2468fb2]: UNREACHABLE! => {"changed": false, "msg": "SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue", "unreachable": true} | 13:51 |
| fungi | AJaeger: thanks for writing that up, a quick skim looks promising but i'll dive into it after coffee | 13:51 |
| odyssey4me | AJaeger yolanda I take it that we're still working through the aftermath of some failures? I'm specifically concerned with how https://review.openstack.org/292326 merged, the sync got executed and we have an empty repo in http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate as a result. :/ | 13:51 |
| AJaeger | fungi, I have that git grep oslo_db |grep tests automated now and can give more - just wanted some first ideas... | 13:52 |
| yolanda | odyssey4me, so several issues seem to be happening. I'm taking a look at puppet failures on gerrit | 13:52 |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box' https://review.openstack.org/286137 | 13:52 |
| yolanda | i'd prefer if another infra-root can help with zuul, i only can see the solution of restarting the service | 13:52 |
| odyssey4me | yolanda thanks, I'll leave you to it - ping me if you come to a resolution or need anything from me | 13:53 |
| *** Jeffrey4l has joined #openstack-infra | 13:53 | |
| *** yamamoto has quit IRC | 13:53 | |
| AJaeger | odyssey4me: coffee, chocolate, ...? ;) | 13:53 |
| fungi | pc_m: coverage jobs are generally very slow. i think it's because they run unit tests serially to accommodate the coverage validation piece. this is a big reason we've previously mostly run coverage post-merge rather than in check/gate (aside from the fact that gating on a specific percentage of coverage is sort of pathological) | 13:53 |
| * odyssey4me hands AJaeger, yolanda, gerrit and zuul a cup of strong coffee :) | 13:54 | |
| thiagop | fungi rcarrillocruz found it! The gerrit user on the common.yaml was with e-mail suffix | 13:54 |
| AJaeger | yummy, odyssey4me ! Thanks | 13:54 |
| rcarrillocruz | :-) | 13:54 |
| *** bdemers has joined #openstack-infra | 13:55 | |
| thiagop | rcarrillocruz: some idiot (aka me) configured it wrong | 13:55 |
| *** bdemers has quit IRC | 13:55 | |
| rcarrillocruz | that message is pretty generic as i said, i saw that with various misconf issues | 13:55 |
| rcarrillocruz | glad you fixed it | 13:55 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Changing gate on devstack identity v3 only voting https://review.openstack.org/292894 | 13:55 |
| thiagop | rcarrillocruz: thank you, and you too fungi | 13:56 |
| rcarrillocruz | np | 13:56 |
| pc_m | fungi: We wanted to do comparison, so that people had an idea of whether their commit was improving coverage or not. | 13:56 |
| * pc_m coverage in the 60% range currently | 13:57 | |
| *** keedya has joined #openstack-infra | 13:58 | |
| yolanda | fungi, checking puppet runs on review.openstack.org, i get fatal: [87741fa9-92cb-4d1e-8195-6841e2468fb2]: FAILED! => {"failed": true, "msg": "'dict object' has no attribute 'gitinfo'"} | 13:59 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Edit task notes from board https://review.openstack.org/291755 | 14:01 |
| pc_m | fungi: on a somewhat related note: is there a way to have py27 run serial fashion in a local repo? | 14:01 |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/system-config: Add vhost for Neutron reviews dashboard https://review.openstack.org/284281 | 14:03 |
| Eyal_ | Is there a problem with the zuul gateway there are no jobs running ? | 14:03 |
| *** Guest41345 has quit IRC | 14:03 | |
| fungi | dhellmann: odd... 2016-03-15 13:27:19,936 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f4f8643a350 292539,3> does not match pipeline requirement <ChangeishFilter open: True current-patchset: True required_approvals: [{'username': 'jenkins', 'verified': [1, 2]}, {'workflow': 1}]> | 14:03 |
| fungi | i disagree with the log, it looks to me at least like those conditions are met | 14:03 |
| AJaeger | Eyal_: yes, we're on it... | 14:04 |
| *** mrmartin has joined #openstack-infra | 14:05 | |
| Eyal_ | AJaeger: thanks | 14:05 |
| SamYaple | X/win 66 | 14:07 |
| *** Eyal_ has quit IRC | 14:07 | |
| *** bdemers has joined #openstack-infra | 14:07 | |
| *** bryan_att has joined #openstack-infra | 14:07 | |
| fungi | okay, i'm now caught up with scrollback and absorbing the additional details of the zuul/gerrit situation | 14:08 |
| *** piet has joined #openstack-infra | 14:08 | |
| *** infra-red has quit IRC | 14:08 | |
| *** armax has joined #openstack-infra | 14:10 | |
| *** smarcet has joined #openstack-infra | 14:10 | |
| *** armax has left #openstack-infra | 14:10 | |
| *** infra-red has joined #openstack-infra | 14:11 | |
| *** salv-orlando has quit IRC | 14:12 | |
| *** salv-orlando has joined #openstack-infra | 14:13 | |
| *** infra-red has quit IRC | 14:13 | |
| *** infra-red has joined #openstack-infra | 14:13 | |
| fungi | i don't see any recent changes to zuul's configuration to account for the current behavior | 14:15 |
| *** ajmiller has joined #openstack-infra | 14:17 | |
| yolanda | fungi so this morning zuul was totally stuck, publishing results. A gerrit restart seemed to unlock it, but then the problems on gate started | 14:17 |
| yolanda | check started to work fine but then no changes were entering gate, and the error was that the change was not ready to be queued | 14:17 |
| fungi | yeah, i read the scrollback | 14:17 |
| *** jlanoux has quit IRC | 14:18 | |
| fungi | zuul's been running since the 11th, so if it's a recently introduced bug then it's been lying dormant for 4 days | 14:18 |
| *** rbrndt has joined #openstack-infra | 14:19 | |
| yolanda | fungi is zuul -t reliable to validate syntax in layout.yaml? | 14:19 |
| yolanda | that's one of the things i thought, that some change introduced in layout.yaml, could have broken it | 14:20 |
| *** oanson has quit IRC | 14:20 | |
| *** esikachev has quit IRC | 14:21 | |
| *** esikachev has joined #openstack-infra | 14:22 | |
| fungi | well, the behavior i posted above looks pretty self-contradictory, so i don't suspect it's something a malformed config could cause | 14:22 |
| AJaeger | yolanda: we run zuul -t as part of our project-config gate, see tools/run-layout.sh | 14:22 |
| *** sshnaidm has quit IRC | 14:23 | |
| fungi | finding a lot of "AttributeError: 'str' object has no attribute '_ZuulGearmanClient__gearman_job'" in the log, but that's been going on since well before the last restart | 14:24 |
| *** sshnaidm has joined #openstack-infra | 14:25 | |
| yolanda | AJaeger, i see. I miss the job-list in that command | 14:25 |
| *** sdake has joined #openstack-infra | 14:25 | |
| *** akshai has quit IRC | 14:25 | |
| yolanda | so that will be the problem i had when trying to test | 14:25 |
| *** ihrachys has quit IRC | 14:25 | |
| fungi | we have reconfiguration failures in the logs, just found one but the traceback is lengthy so will put in a paste in a moment | 14:25 |
| *** mrmartin has quit IRC | 14:26 | |
| *** akshai has joined #openstack-infra | 14:26 | |
| fungi | yolanda: exception raised during zuul reconfiguration http://paste.openstack.org/show/490518/ | 14:27 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add playbook to manage power state of ironic servers https://review.openstack.org/284981 | 14:27 |
| fungi | looks like maybe we merged a bad regular expression? | 14:27 |
| *** infra-red has quit IRC | 14:27 | |
| yolanda | mm, that's error i got when running the zuul-server -t as well | 14:27 |
| *** infra-red has joined #openstack-infra | 14:28 | |
| fungi | have we maybe broken the exit code for our zuul configuration check jobs? | 14:28 |
| *** fhubik has quit IRC | 14:29 | |
| *** fhubik has joined #openstack-infra | 14:29 | |
| yolanda | i'm checking the changes that landed this morning | 14:29 |
| fungi | first occurrence was at 08:23:21 utc today | 14:29 |
| fungi | so it would have been something merging just prior to that | 14:30 |
| fungi | there were a couple this morning with regex modifications | 14:30 |
| fungi | one is fairly straightforward, the other less so | 14:30 |
| yolanda | do you have some candidates? | 14:30 |
| *** Guest91794 has quit IRC | 14:31 | |
| fungi | https://review.openstack.org/292187 | 14:31 |
| yolanda | that regex looks fine... | 14:31 |
| smarcet | fungi: yolanda: morning guys, if u have time could u please review https://review.openstack.org/#/c/292605/? i need this to add error email log capabilities | 14:31 |
| yolanda | the doc not ending with $ ? | 14:31 |
| *** fhubik has quit IRC | 14:31 | |
| *** fhubik_real has joined #openstack-infra | 14:32 | |
| fungi | i'm having a look at the job results to see what our layout check said | 14:32 |
| fungi | but i haven't moved to my main computer for the day, and loading the console log for that job is taking forever on this very under-powered netbook | 14:33 |
| AJaeger | smarcet: right now nothing can merge, we first need to debug and fix our CI system, please have some patience | 14:33 |
| yolanda | yep, i was loading the check for layout.yaml as well, but is long to see | 14:33 |
| *** kingia_ has joined #openstack-infra | 14:33 | |
| *** xyang1 has joined #openstack-infra | 14:33 | |
| yolanda | there is a traceback on apscheduler, <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'now' . But that looks unrelated. And test reports no errors found | 14:34 |
| fungi | "No errors found in layout.yaml!" | 14:34 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Refactoring https://review.openstack.org/292912 | 14:34 |
| *** salv-orlando has quit IRC | 14:35 | |
| *** salv-orl_ has joined #openstack-infra | 14:35 | |
| yolanda | well, all these tests have been positive.. so if there is an error, is hidden somehow | 14:35 |
| *** kingia has quit IRC | 14:35 | |
| fungi | is your reproduction with zuul -t on your local system? | 14:36 |
| yolanda | no, not local | 14:36 |
| fungi | oh, not as easy to bisect then | 14:36 |
| *** bdemers has quit IRC | 14:37 | |
| *** ihrachys has joined #openstack-infra | 14:37 | |
| yolanda | fungi, i can test locally, copying the layout.yaml | 14:37 |
| yolanda | and i can remove the suspectingo nes | 14:38 |
| jeblair | o/ | 14:38 |
| *** kingia has joined #openstack-infra | 14:38 | |
| fungi | yeah, if you can get it to fail on the current state of layout.yaml, then try before today's changes merged for a baseline | 14:39 |
| fungi | jeblair: summary is that zuul is claiming changes which should be ready to merge are not matching the required criteria | 14:39 |
| fungi | jeblair: we also have reconfiguration exceptions in the log starting at 08:23:21 utc today and the traceback at http://paste.openstack.org/show/490518/ makes me suspect we merged a bad regex | 14:40 |
| fungi | i don't know if the two issues are related, but have a feeling they might be | 14:41 |
| yolanda | fungi, we have a winner | 14:41 |
| *** mtanino has joined #openstack-infra | 14:41 | |
| yolanda | https://review.openstack.org/#/c/292187 | 14:41 |
| fungi | yolanda: was it 292187? | 14:41 |
| fungi | indeed | 14:41 |
| yolanda | yes | 14:41 |
| fungi | so i guess that needs an emergency fix or revert | 14:41 |
| fungi | followed by a manual submit in gerrit | 14:42 |
| *** kingia_ has quit IRC | 14:42 | |
| yolanda | i'm checking the problem, i spotted a missing $ in docs, but not sure if that's it | 14:42 |
| *** ddieterly has joined #openstack-infra | 14:42 | |
| fungi | also we need to understand why zuul -t fails for you on it but doesn't fail in the layout job which passed that change with flying colors | 14:42 |
| yolanda | problem is on the name of the project, that regex fails | 14:44 |
| yolanda | but yes, is risky that the check job for that passes | 14:44 |
| *** yamahata has quit IRC | 14:44 | |
| *** mrmartin has joined #openstack-infra | 14:44 | |
| yolanda | fungi, how do you operate on these cases? in downstream i would have stopped puppet, fix layout.yaml manually | 14:44 |
| *** keedya has quit IRC | 14:44 | |
| yolanda | so zuul starts running, then do the revert | 14:45 |
| *** mrmartin has quit IRC | 14:45 | |
| yolanda | and restore puppet | 14:45 |
| *** zz_dimtruck is now known as dimtruck | 14:45 | |
| cloudnull | o/ all -- question, did our gerrit sign-in change to a new openID solution instead of the Launchpad one? | 14:45 |
| * cloudnull is likely out of the loop | 14:45 | |
| mriedem | AJaeger: question in here for you on the different nodes https://review.openstack.org/#/c/289255/2/jenkins/jobs/projects.yaml | 14:45 |
| fungi | yolanda: in cases like this it's almost as quick to just shove it through gerrit and update puppet out of cycle | 14:45 |
| fungi | yolanda: but if you want to stop puppet and fix it by hand instead go ahead | 14:46 |
| yolanda | so revert and force push? do we have the right perms for that normally, or is something that we need to add? | 14:46 |
| cloudnull | im seeing http://snag.gy/AzqwE.jpg right now. | 14:46 |
| fungi | just don't forget to unblock puppet for it again afterward | 14:46 |
| yolanda | ok going to do it with stopping puppet... | 14:47 |
| *** regXboi has joined #openstack-infra | 14:47 | |
| jeblair | yolanda: don't force push; there's a group called "Project Bootstrappers"; you can add yourself to it, then use the gerrit ui to submit the change manually | 14:47 |
| fungi | yolanda: no, you can put it into gerrit as a normal review, add yourself to the project-bootstrappers group in gerrit, code review +2 verify +2 workflow +1 it and then submit it from the change screen | 14:47 |
| *** Eyal_ has joined #openstack-infra | 14:48 | |
| fungi | cloudnull: we haven't changed the gerrit openid provider, no. should still be login.launchpad.net | 14:48 |
| AJaeger | mriedem: answered briefly. Does that help? | 14:48 |
| jeblair | fungi, cloudnull: i get that error too | 14:48 |
| mriedem | AJaeger: so should he use ubuntu-trusty now? | 14:48 |
| mordred | I also get the error | 14:48 |
| AJaeger | mriedem: Yes! | 14:48 |
| mordred | has login.launchpad.net been sunsetted for login.ubuntu.com finally? | 14:48 |
| mriedem | AJaeger: thanks | 14:49 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Revert "Do not run gate jobs when doing docs-only patch" https://review.openstack.org/292924 | 14:49 |
| mordred | login.lp.net/+openid is not loading for me | 14:49 |
| yolanda | infra-root so can you review that? | 14:49 |
| *** Eyal_ has quit IRC | 14:49 | |
| cloudnull | seems LP is having issues http://cdn.pasteraw.com/e5kxzqy1enz9q0a0qe5dy6wug256298 | 14:49 |
| yolanda | jeblair ++ | 14:50 |
| cloudnull | http://snag.gy/C0v3f.jpg | 14:50 |
| mordred | login.ubuntu.com is also not loading, fwiw | 14:50 |
| jeblair | clicking on 'log in' on launchpad is hanging as well | 14:51 |
| fungi | and yeah, i suspect the line ending inside the negative lookahead on that project name regex is the issue | 14:51 |
| fungi | i can probably add myself to project-bootstrappers via the gerrit api | 14:51 |
| cloudnull | from #launchpad <cjwatson> frederickjh: Yes, there are issues with one of our internal infrastructure clouds; it's being worked on. | 14:52 |
| yolanda | ouch.. Provider is not supported, or was incorrectly entered. | 14:52 |
| mordred | cloudnull: awesome. thanks | 14:52 |
| yolanda | when trying to login to gerrit | 14:52 |
| mordred | jeblair, fungi: seems like a good topic for an alert | 14:52 |
| jeblair | mordred: ++; you typing it? | 14:53 |
| fungi | agreed, go for it. i'm messing with gerrit's set-group-members function | 14:53 |
| *** dosaboy has quit IRC | 14:53 | |
| mordred | jeblair: yah - is it #alert or #status (I'm still coffeeing)_ | 14:53 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Edit task notes from board https://review.openstack.org/291755 | 14:53 |
| jeblair | mordred: "#status alert <text>" | 14:54 |
| *** maishsk has quit IRC | 14:54 | |
| *** dosaboy has joined #openstack-infra | 14:54 | |
| *** dizquierdo has joined #openstack-infra | 14:54 | |
| *** hichihara has quit IRC | 14:54 | |
| mordred | how about: #status alert Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue | 14:54 |
| jeblair | lgtm | 14:55 |
| mordred | #status alert Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue | 14:55 |
| openstackstatus | mordred: sending alert | 14:55 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Do not run gate jobs when doing docs-only patch" https://review.openstack.org/292924 | 14:55 |
| *** dosaboy has quit IRC | 14:55 | |
| jeblair | fungi: nicely done! | 14:55 |
| fungi | webui-less! | 14:55 |
| fungi | now to take myself back out of the p-b group again | 14:56 |
| mfisch | hey infra folks, openid signin is broken/looks really weird | 14:56 |
| *** dosaboy has joined #openstack-infra | 14:56 | |
| mfisch | "Provider is not supported, or was incorrectly entered." | 14:56 |
| *** maishsk has joined #openstack-infra | 14:56 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 14:56 |
| *** dosaboy has quit IRC | 14:57 | |
| AJaeger | mfisch: see status alert that is just send around | 14:57 |
| fungi | mfisch: irc channel topics are updating as we speak | 14:57 |
| -openstackstatus- NOTICE: Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue | 14:57 | |
| mfisch | im always missing the hot trends | 14:57 |
| *** ChanServ changes topic to "Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue" | 14:57 | |
| fungi | there you go | 14:57 |
| mfisch | ty | 14:58 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: The puppet-cookiecutter need gems to works. https://review.openstack.org/292933 | 14:58 |
| yolanda | infra-root, is there any way to force a zuul reconfig? or just wait? | 14:58 |
| fungi | yolanda: you can manually puppet apply if you want to just exercise the machinery faster | 14:59 |
| fungi | instructions are in our system-config docs | 14:59 |
| *** links has quit IRC | 14:59 | |
| SamYaple | hello, recently merged this jobs filter for kolla with skip-if https://review.openstack.org/#/c/292187/, it _should_ be filtering all gates except docs when a docs patch is detected but it is not doing so see https://review.openstack.org/#/c/292869/ | 14:59 |
| *** jpr1 has quit IRC | 14:59 | |
| SamYaple | does this filter take a little while to take affect or is it incorrect | 14:59 |
| yolanda | fungi but the update of layout.yaml is triggering the reload of zuul automatically? | 14:59 |
| yolanda | SamYaple, we just reverted the change | 14:59 |
| fungi | SamYaple: reverted in https://review.openstack.org/292924 | 15:00 |
| *** markvoelker has quit IRC | 15:00 | |
| *** jpr has joined #openstack-infra | 15:00 | |
| clarkb | yolanda: no, you have to update the file then appl a signal to zuul. Puppet does both | 15:00 |
| fungi | problem with the regex somehow isn't spotted by the check job but blew up zuul all morning | 15:00 |
| openstackstatus | mordred: finished sending alert | 15:00 |
| yolanda | clarkb ok good to know | 15:00 |
| *** mrmartin has joined #openstack-infra | 15:01 | |
| tobiash | where does zuul fill in the reporter_config? | 15:01 |
| *** jaosorior has quit IRC | 15:01 | |
| pabelanger | anteaya: clarkb: Sorry, was on PTO yesterday. | 15:01 |
| SamYaple | so are you saying i broke things yolanda fungi ? | 15:01 |
| jeblair | can we move to incident? | 15:01 |
| tobiash | my gerrit rejects "--verified 2" in the gate pipeline where "--label verified=2" seems to work somehow | 15:02 |
| *** roxanagh_ has joined #openstack-infra | 15:02 | |
| pabelanger | clarkb: I'd have to look into grafana and see how to cache the data. I know you can have graphite render the image a static over grafana using css | 15:02 |
| SamYaple | ah man. thats awful! I feel bad now | 15:02 |
| *** rhedlind has joined #openstack-infra | 15:03 | |
| SamYaple | whats wrong with that regex, it validates for me what I would expect it to do | 15:03 |
| fungi | SamYaple: don't feel bad. we obviously have a test coverage issue this exposed (as well as people approving regular expressions without trying them out) | 15:03 |
| pabelanger | anteaya: re: stackalytics.o.o no, not yet. We needed to enable logs first to see what is happening. It would be good to compare to stackalytics.com too, and see if they are pinning the CPU | 15:03 |
| SamYaple | fungi: well it does work for me with POSIX grep, but is it evaluated differently in the gate? | 15:04 |
| fungi | SamYaple: http://paste.openstack.org/show/490518/ is the traceback | 15:04 |
| SamYaple | esh. ok let me test this out in python | 15:05 |
| SamYaple | i didnt know how it was being eval'd, but re makes sense | 15:05 |
| SamYaple | so the gate job isn't processing the regex. that seems odd | 15:06 |
| SamYaple | i think it failed on my first patchset | 15:06 |
| *** tesseract has joined #openstack-infra | 15:06 | |
| *** bdemers has joined #openstack-infra | 15:06 | |
| *** adrian_otto has joined #openstack-infra | 15:07 | |
| *** dosaboy has joined #openstack-infra | 15:07 | |
| *** roxanagh_ has quit IRC | 15:07 | |
| *** tesseract is now known as Guest71467 | 15:07 | |
| adrian_otto | is there a known problem with SSO for review.openstack.org? | 15:07 |
| SamYaple | adrian_otto: cloudnull was just talking about that | 15:07 |
| SamYaple | 14:53:20 < cloudnull> from #launchpad <cjwatson> frederickjh: Yes, there are issues with one of our internal infrastructure clouds; it's being worked on. | 15:08 |
| SamYaple | adrian_otto: ^ | 15:08 |
| adrian_otto | thanks SamYaple | 15:08 |
| *** IlyaG has joined #openstack-infra | 15:09 | |
| clarkb | we also updated all the channel topics with info on that | 15:09 |
| *** dosaboy has quit IRC | 15:09 | |
| adrian_otto | thanks clarkb | 15:09 |
| SamYaple | fungi: regex seems valid.... http://paste.openstack.org/show/490527/ | 15:10 |
| dmsimard | pabelanger: how are you setting up/collecting the various roles you need to run playbooks right now ? Galaxy or setup.py ? | 15:10 |
| dmsimard | i.e, your dib or zuul roles | 15:10 |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add Ubuntu binary build and deploy gate for kolla https://review.openstack.org/292942 | 15:12 |
| *** IlyaG has quit IRC | 15:13 | |
| *** krtaylor has quit IRC | 15:13 | |
| pabelanger | dmsimard: zuul-cloner right now | 15:14 |
| pabelanger | dmsimard: https://github.com/openstack/windmill/blob/master/tools/install_roles.sh | 15:15 |
| SamYaple | fungi: is there more info we can get on that traceback? As far as i can tell that regex is valid (and local python re agrees with me) so i wonder what happens in translation from yaml to it being compiled | 15:16 |
| pabelanger | I've started working on a script to parse from ansible requirment.yml file to zuul-cloner but still need to finish it up | 15:16 |
| dmsimard | pabelanger: I've never published anything ansible-galaxy ready, trying to look into it right now | 15:17 |
| fungi | SamYaple: yeah, we're troubleshooting over in #openstack-infra-incident so as not to get drowned out by the general support noise in here | 15:17 |
| fungi | SamYaple: i'm trying to recreate the failure which generates that traceback | 15:17 |
| pabelanger | dmsimard: well, ansible-galaxy publish, is just them reading your github.com repo and cloning it locally | 15:17 |
| pabelanger | I tried to use it a few weeks ago, but it was forcing a README.md file | 15:18 |
| pabelanger | dmsimard: pretty sure they fixed it to use README.rst but haven't tested yet | 15:18 |
| dmsimard | pabelanger: yeah, I've been meaning to try and use the requirements.yml file with galaxy | 15:18 |
| dmsimard | not necessarily publishing | 15:18 |
| dmsimard | I thought you needed something like a metadata.json file (akin to puppet) but nothing fancy like that, no. | 15:19 |
| *** _amrith_ is now known as amrith | 15:19 | |
| pabelanger | dmsimard: Ya, i've tested it locally and it works great | 15:19 |
| pabelanger | just like zuul-cloner better :) | 15:19 |
| dmsimard | well, in your context it makes sense | 15:19 |
| *** dosaboy has joined #openstack-infra | 15:19 | |
| pabelanger | agreed | 15:20 |
| *** piet has quit IRC | 15:20 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 15:23 |
| *** yamahata has joined #openstack-infra | 15:24 | |
| yolanda | ok zuul seems to be back to life | 15:24 |
| *** infra-red has quit IRC | 15:24 | |
| *** krtaylor has joined #openstack-infra | 15:25 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Make task notes visible for logged-out users https://review.openstack.org/292957 | 15:26 |
| *** salv-orl_ has quit IRC | 15:27 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 15:27 |
| *** pvaneck has joined #openstack-infra | 15:27 | |
| mordred | infra-root: launchpad is back- I'm going to send the ok | 15:27 |
| jeblair | mordred: thanks; remember you can add text after the ok "#status ok <text>" | 15:28 |
| mordred | #status ok Launchpad SSO is back to normal - happy hacking | 15:28 |
| openstackstatus | mordred: sending ok | 15:28 |
| mordred | jeblair: thanks! I'd forgotten that | 15:28 |
| *** infra-red has joined #openstack-infra | 15:28 | |
| *** armax has joined #openstack-infra | 15:29 | |
| *** MarkAtwood has joined #openstack-infra | 15:29 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 15:30 | |
| -openstackstatus- NOTICE: Launchpad SSO is back to normal - happy hacking | 15:30 | |
| yolanda | zuul happy, launchpad happy, we are happy now :) | 15:31 |
| yolanda | AJaeger, have the projects been created? i ran manage-projects on gerrit | 15:31 |
| *** xyang1 has quit IRC | 15:31 | |
| pabelanger | And, I'm offline again for the next hour or so. Relocating back to relatives farm. Then I'm back for the rest of the day | 15:32 |
| openstackstatus | mordred: finished sending ok | 15:33 |
| *** chlong has quit IRC | 15:33 | |
| *** jlanoux has joined #openstack-infra | 15:34 | |
| *** nmagnezi has quit IRC | 15:35 | |
| *** infra-red has quit IRC | 15:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Start adding test for bifrost dhcp https://review.openstack.org/292521 | 15:37 |
| *** Qiming has quit IRC | 15:38 | |
| *** fawadkhaliq has quit IRC | 15:40 | |
| *** fawadkhaliq has joined #openstack-infra | 15:40 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Rename the ceph to upgrades https://review.openstack.org/292359 | 15:41 |
| *** Swami has joined #openstack-infra | 15:41 | |
| *** kushal has joined #openstack-infra | 15:42 | |
| *** sridhar_ram has joined #openstack-infra | 15:43 | |
| *** ddieterly is now known as ddieterly[away] | 15:43 | |
| *** infra-red has joined #openstack-infra | 15:43 | |
| *** vgridnev has quit IRC | 15:43 | |
| *** maishsk has quit IRC | 15:43 | |
| *** fawadkhaliq has quit IRC | 15:45 | |
| *** yamahata has quit IRC | 15:45 | |
| *** ddieterly[away] is now known as ddieterly | 15:46 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 15:46 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics. https://review.openstack.org/291392 | 15:46 |
| *** vgridnev has joined #openstack-infra | 15:46 | |
| *** jpr has quit IRC | 15:48 | |
| *** amrith is now known as _amrith_ | 15:49 | |
| *** vgridnev has quit IRC | 15:49 | |
| *** Jeffrey4l has quit IRC | 15:50 | |
| *** infra-red has quit IRC | 15:50 | |
| *** tongli has joined #openstack-infra | 15:51 | |
| *** gyee has joined #openstack-infra | 15:51 | |
| *** acruz has joined #openstack-infra | 15:51 | |
| *** daemontool has joined #openstack-infra | 15:51 | |
| bartash | hi all | 15:52 |
| *** arxcruz has quit IRC | 15:52 | |
| clarkb | bartash: hello | 15:52 |
| *** maishsk has joined #openstack-infra | 15:53 | |
| bartash | could anyone help with the issue on this review? https://review.openstack.org/#/c/292302/ jenkins fails to execute py27 job | 15:54 |
| clarkb | bartash: looks like it executes, it just fails to succeed. See http://logs.openstack.org/02/292302/1/check/gate-shaker-python27/9b8f5d0/console.html#_2016-03-14_11_38_21_828 for the error | 15:55 |
| clarkb | bartash: looks like a missing dependency | 15:55 |
| clarkb | bartash: I want to say the the pyzmq lib would bundle the C and compile it on install if it had to. You will probably want to double check on that, maybe they broke their packaging | 15:56 |
| *** bpokorny has joined #openstack-infra | 15:56 | |
| *** nwkarsten has quit IRC | 15:57 | |
| *** asselin__ has joined #openstack-infra | 15:57 | |
| *** salv-orlando has joined #openstack-infra | 15:57 | |
| *** asselin_ has quit IRC | 15:57 | |
| *** links has joined #openstack-infra | 15:58 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu https://review.openstack.org/293012 | 15:58 |
| bartash | clarkb: yes, I saw that log file too. But how to solve this problem and whom should I ask about it? | 15:59 |
| *** IlyaG has joined #openstack-infra | 15:59 | |
| clarkb | bartash: the first thing I would do is try to reproduce it locally. Make sure you rebuild your virtualenv when you do `tox -repy27` | 15:59 |
| clarkb | bartash: then go from there, chase it up the pyzmq tree and see why that is breaking | 15:59 |
| *** acruz has quit IRC | 16:00 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Move layout job to bare-precise https://review.openstack.org/293015 | 16:00 |
| *** akshai has quit IRC | 16:00 | |
| *** akshai has joined #openstack-infra | 16:01 | |
| *** mrmartin has quit IRC | 16:02 | |
| bartash | clarkb: it succeeded on my local virtualenv | 16:03 |
| *** maishsk has quit IRC | 16:03 | |
| *** asettle has joined #openstack-infra | 16:03 | |
| bartash | clarkb: you mean go through the console log with that failure? | 16:03 |
| clarkb | bartash: did you rebuild with tox -r? | 16:03 |
| *** _amrith_ is now known as amrith | 16:04 | |
| clarkb | bartash: I think it is possibly a new packaging issue | 16:04 |
| clarkb | so you want to rebuild | 16:04 |
| *** tphummel has joined #openstack-infra | 16:06 | |
| morgan | clarkb, seeing something weird in gerrit - i have a change that is +2/+A but the list view isn't reflecting it. | 16:06 |
| *** _david_ has quit IRC | 16:06 | |
| clarkb | bartash: it is using an older pyzmq version according to the freeze | 16:06 |
| morgan | clarkb: not sure if it's something out of whack or a weird cache thing. | 16:07 |
| morgan | or something wider spread. | 16:07 |
| fungi | morgan: which change? | 16:07 |
| clarkb | bartash: possibly due to the constraints? | 16:07 |
| fungi | we're trying to hunt down one for further testing | 16:07 |
| morgan | fungi: so looking at https://review.openstack.org/#/q/project:openstack/stackalytics+status:open https://review.openstack.org/#/c/292653/ | 16:07 |
| morgan | fungi: happy to help find one :) | 16:07 |
| *** asettle has quit IRC | 16:08 | |
| *** ramishra has quit IRC | 16:08 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Kolla docs-only patch -- take 2 https://review.openstack.org/293021 | 16:08 |
| *** bpokorny has quit IRC | 16:09 | |
| *** bpokorny has joined #openstack-infra | 16:10 | |
| *** ramishra has joined #openstack-infra | 16:10 | |
| bartash | clarkb: I'm quite noob in openstack gerrit and don't know about how this all mechanism works inside. We just depend on the patch shown above, and I wanted to find someone who can solve it, as it looks like a problem is not connected to Shaker project itself, isn't it? | 16:11 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the container CI job https://review.openstack.org/293022 | 16:11 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate https://review.openstack.org/293023 | 16:11 |
| clarkb | bartash: it is a problem in the shaker test suite | 16:11 |
| *** ddieterly is now known as ddieterly[away] | 16:11 | |
| clarkb | bartash: it relies on pyzmq but it isn't getting installed properly. You can find the install logs at http://logs.openstack.org/02/292302/1/check/gate-shaker-python27/9b8f5d0/tox/ for that job | 16:12 |
| bartash | clarkb: ok, I understand. thank you a lot for help. | 16:13 |
| clarkb | bartash: to clarify, gerrit is just a code review tool. Connected to gerrit we have zuul which runs test jobs on changes. This is one of those tests jobs and it runs `tox -repy27` and that was the result | 16:13 |
| bartash | clarkb: so the problem is just in shaker-job, right? | 16:13 |
| *** cbader has quit IRC | 16:13 | |
| *** cbader has joined #openstack-infra | 16:14 | |
| clarkb | bartash: yes that is where I would start. tox -repy27 is returning an error so that is what needs to be debugged | 16:14 |
| *** sc68cal has joined #openstack-infra | 16:14 | |
| *** mrmartin has joined #openstack-infra | 16:14 | |
| bartash | clarkb: ok, thanks, I'll continue looking in this direction | 16:15 |
| fungi | morgan: the approval event for that change seems to have been missed by zuul during an event stream issue earlier today. it'll need to be reapproved | 16:16 |
| *** armax has quit IRC | 16:16 | |
| morgan | fungi: right. is that related to why the list view doesn't reflect the detail view? | 16:16 |
| morgan | fungi: i was less concerned about it needing reapproval vs something wrong with gerrit :) | 16:16 |
| fungi | morgan: oh, i was looking at it in gertty and didn't see anything odd. i'll check with the webui | 16:17 |
| morgan | fungi: yeah webui is inconsistent even with hard refreshes. | 16:18 |
| *** kzaitsev_mb has quit IRC | 16:18 | |
| fungi | morgan: i don't spend enough time looking at the gerrit webui... what specifically is missing? what are list view and detail view? | 16:19 |
| morgan | fungi: the +2 / +A isn't showing in the list view | 16:19 |
| morgan | fungi: where it shows in the detail view | 16:19 |
| *** amrith is now known as _amrith_ | 16:19 | |
| fungi | ahh! on your query | 16:19 |
| fungi | the query result list view. got it | 16:19 |
| morgan | yep | 16:20 |
| *** ramishra has quit IRC | 16:21 | |
| *** hashar has quit IRC | 16:21 | |
| fungi | morgan: that is indeed a strange inconsistency. i wonder if we have something out of sync in gerrit's database | 16:22 |
| fungi | or the lucene index | 16:22 |
| *** roxanagh_ has joined #openstack-infra | 16:22 | |
| morgan | fungi: yah | 16:22 |
| *** ramishra has joined #openstack-infra | 16:23 | |
| *** tzn has quit IRC | 16:23 | |
| *** tzn has joined #openstack-infra | 16:24 | |
| *** markvoelker has joined #openstack-infra | 16:25 | |
| *** jed56 has joined #openstack-infra | 16:27 | |
| fungi | morgan: aha, the approval was at 10:26 utc and the start time on our gerrit process (from this morning's restart) was also 10:26 utc. i wonder if there's a race in gerrit which could account for that specific change | 16:28 |
| morgan | fungi: maybe. happy to get it a re-freshed approval | 16:28 |
| morgan | fungi: if you're content no issue(s) i'll go do that/recheck/whatever | 16:29 |
| morgan | fungi: but it looked weird so wanted to pass on first :) | 16:29 |
| sc` | i had to -/+ workflow a change to get it picked up after i had already approved it | 16:29 |
| *** tzn has quit IRC | 16:29 | |
| *** ramishra has quit IRC | 16:30 | |
| sc` | reverify was just kicking off recheck it appeared | 16:30 |
| *** mikelk has quit IRC | 16:31 | |
| jeblair | sc`: yes, that's a known quirk of the system used to ensure that patches pass check before being enqueued in gate | 16:31 |
| *** ramishra has joined #openstack-infra | 16:31 | |
| *** sdake_ has joined #openstack-infra | 16:31 | |
| *** roxanagh_ has quit IRC | 16:32 | |
| sc` | good to know | 16:32 |
| clarkb | depending on how the periodic jobs go/went https://review.openstack.org/#/c/292573/ would be a good one to get in to finish up the devstack-trusty to ubuntu-trusty migration | 16:32 |
| * clarkb looks for evidence of goodness on ubuntu-trusty | 16:34 | |
| *** sdake has quit IRC | 16:34 | |
| *** esikachev has quit IRC | 16:35 | |
| *** rcernin has quit IRC | 16:35 | |
| *** skolekonov is now known as _skolekonov | 16:36 | |
| clarkb | http://logs.openstack.org/41/285541/5/check/gate-tempest-dsvm-full-liberty/93061ef/console.html fails but looks like a legit tempest fail not host related | 16:36 |
| *** flongo has quit IRC | 16:36 | |
| *** eil397 has joined #openstack-infra | 16:38 | |
| *** andrearosa has quit IRC | 16:38 | |
| *** tmorin has joined #openstack-infra | 16:39 | |
| clarkb | jenkins07 has almost 7k threads I am going to run the restart playbook again it | 16:42 |
| *** Guest71467 has quit IRC | 16:42 | |
| *** piet has joined #openstack-infra | 16:43 | |
| *** scheuran has quit IRC | 16:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:44 | |
| *** jpr has joined #openstack-infra | 16:45 | |
| *** links has quit IRC | 16:47 | |
| *** tzn has joined #openstack-infra | 16:47 | |
| *** mrmartin has quit IRC | 16:50 | |
| *** apoorvad has joined #openstack-infra | 16:51 | |
| *** vincentll has quit IRC | 16:51 | |
| *** flongo has joined #openstack-infra | 16:52 | |
| *** SumitNaiksatam has quit IRC | 16:52 | |
| anteaya | pabelanger: okay happy relocating | 16:53 |
| *** tzn has quit IRC | 16:54 | |
| *** dprince has quit IRC | 16:54 | |
| *** MCoLo__ has quit IRC | 16:54 | |
| tmorin | hi sdague: would you have some time to look at https://review.openstack.org/#/c/291103/ | 16:55 |
| *** akshai has quit IRC | 16:55 | |
| tmorin | tmorin: it's a simple job change to move job parameters from the job yaml file to an rc file in the project repo | 16:55 |
| tmorin | AJaeger wanted you to have a look | 16:56 |
| tmorin | and it was already blessed by dougwig as Neutron liaison | 16:56 |
| *** nwkarsten has joined #openstack-infra | 16:57 | |
| *** akshai has joined #openstack-infra | 16:57 | |
| *** vgridnev has joined #openstack-infra | 16:58 | |
| *** akshai has quit IRC | 16:58 | |
| *** e0ne has quit IRC | 16:58 | |
| *** sc68cal has quit IRC | 16:59 | |
| jeblair | tmorin: is that file in every branch of the repo? what's the plan for managing changes to that across multiple branches? | 17:00 |
| *** akshai has joined #openstack-infra | 17:00 | |
| *** fawadkhaliq has joined #openstack-infra | 17:00 | |
| tmorin | jeblair: no, its not in every branch | 17:01 |
| tmorin | jeblair: the jobs are experimental only, we plan to enable them only in recent branches | 17:01 |
| *** nwkarsten has quit IRC | 17:01 | |
| tmorin | jeblair: so we've no plan to put the rcfiles in past branches | 17:01 |
| *** esikachev has joined #openstack-infra | 17:02 | |
| jeblair | fungi, clarkb, yolanda: upon further inspection, a failed reconfiguration should not cause a problem with timer triggers. the code i was looking at is just a bunch of unimplemented methods added by the connections change. i think i have already removed them in zuulv3. | 17:05 |
| *** esikachev has quit IRC | 17:07 | |
| jeblair | fungi, clarkb, yolanda: a failed reconfiguration will cause some kind of strange state with the scheduler losing track of the triggers it has defined, though i can't actually figure out what the result of that would be. i think they will continue to work. | 17:07 |
| yolanda | jeblair, i wonder if there is some way to prevent these reconfiguration failures or make them more visible on logs | 17:08 |
| jeblair | fungi, clarkb, yolanda: so, as best as i can tell, the code has latent bugs, but they should not actually cause things to stop working. | 17:08 |
| jeblair | yolanda: the reconfiguration failure was prevented -- it never loaded the bad configuration | 17:08 |
| clarkb | jeblair: so the regex not being old python stdlib compatible was the only real problem? | 17:08 |
| *** tqtran has joined #openstack-infra | 17:09 | |
| jeblair | clarkb: the only thing i can prove at this point is that the regex caused the reconfiguration to abort, and zuul continued to use the old config as designed | 17:09 |
| yolanda | infra-root, can there be a problem with gerritbot? i sent a change right now and didn't see anything posted | 17:10 |
| *** nwkarsten has joined #openstack-infra | 17:10 | |
| *** fawadkhaliq has quit IRC | 17:10 | |
| clarkb | yolanda: there are two openstackgerrit's here | 17:10 |
| clarkb | likely something is unhappy | 17:10 |
| *** MarkAtwood has quit IRC | 17:11 | |
| *** asselin__ has quit IRC | 17:11 | |
| *** vgridnev has quit IRC | 17:12 | |
| *** bdemers has quit IRC | 17:12 | |
| *** sambetts is now known as sambetts|afk | 17:12 | |
| yolanda | i see some errors on gerritbot log | 17:13 |
| *** bdemers has joined #openstack-infra | 17:13 | |
| bknudson | AJaeger: does devstack support opensuse? I tried downloading opensuse and it failed with an error | 17:14 |
| anteaya | AJaeger: I figured you would probably have some experience with bknudson's question | 17:14 |
| *** openstackgerrit has quit IRC | 17:14 | |
| *** openstackgerrit_ is now known as openstackgerrit | 17:14 | |
| *** tmorin has left #openstack-infra | 17:14 | |
| *** openstackgerrit has quit IRC | 17:15 | |
| yolanda | restarted gerritbot | 17:15 |
| anteaya | yolanda: thank you | 17:15 |
| *** openstackgerrit_ has joined #openstack-infra | 17:15 | |
| *** openstackgerrit_ is now known as openstackgerrit | 17:16 | |
| *** esikachev has joined #openstack-infra | 17:16 | |
| *** openstackgerrit_ has joined #openstack-infra | 17:16 | |
| fungi | yolanda: clarkb: the extra gerritbot is coming from the replacement gerrit server. could stand to shut that one down for now | 17:17 |
| yolanda | oh i can stop it yes | 17:17 |
| *** fawadkhaliq has joined #openstack-infra | 17:18 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Refactoring https://review.openstack.org/292912 | 17:20 |
| anteaya | well it works | 17:20 |
| anteaya | we can check that off the list for April 11th | 17:20 |
| rbradfor | I'm getting ERROR: missing Change-Id in commit message footer when I'm trying to submit a patchset (3 reviews). I've done this before without any issues.http://paste.openstack.org/show/490559/ | 17:21 |
| *** asettle has joined #openstack-infra | 17:21 | |
| clarkb | rbradfor: what does `git log -1` show? | 17:22 |
| clarkb | er might need -3 based on that short log | 17:22 |
| rbradfor | clarkb, sure -3 show my 3 commits, but no Change-Id | 17:22 |
| clarkb | rbradfor: ok, then you need to force the commit hook to run | 17:23 |
| fungi | yeah, git-review had never been used in that clone before according to the output | 17:23 |
| clarkb | git rebase -i HEAD~3 | 17:23 |
| fungi | so it downloaded the commit hook which adds the change-id after you committed your changes | 17:23 |
| clarkb | change each pick to reword, save each one and that should add teh change ids | 17:23 |
| *** mrmartin has joined #openstack-infra | 17:24 | |
| *** pcaruana has quit IRC | 17:24 | |
| rbradfor | clarkb, fungi thanks, reword has given me change-ids | 17:24 |
| *** SumitNaiksatam has joined #openstack-infra | 17:25 | |
| *** fawadkhaliq has quit IRC | 17:25 | |
| fungi | and it asks nothing in return! | 17:25 |
| clarkb | git-review does its best to add the change id to the last commit if it is missing but doesn't know how to add them to a whole stack | 17:25 |
| *** ddieterly[away] is now known as ddieterly | 17:25 | |
| *** fawadkhaliq has joined #openstack-infra | 17:25 | |
| *** asettle has quit IRC | 17:26 | |
| fungi | yep | 17:27 |
| *** jlanoux has quit IRC | 17:27 | |
| *** jlanoux has joined #openstack-infra | 17:28 | |
| *** jlanoux has quit IRC | 17:28 | |
| *** jlanoux has joined #openstack-infra | 17:29 | |
| *** esikachev has quit IRC | 17:31 | |
| *** matrohon has quit IRC | 17:31 | |
| mriedem | i need some help with what i thought was a simple change | 17:33 |
| mriedem | https://review.openstack.org/#/c/292617/1/zuul/layout.yaml | 17:33 |
| mriedem | easy peasy | 17:33 |
| mriedem | but fails with http://logs.openstack.org/17/292617/1/check/gate-project-config-layout/2be8620/console.html#_2016-03-14_21_59_01_546 | 17:33 |
| clarkb | mriedem: looking. Also we should just merge that es status page change in e-r if it isn't already | 17:33 |
| *** dprince has joined #openstack-infra | 17:33 | |
| mriedem | ^ follows the same pattern as the other jobs using that template from what i can tell | 17:33 |
| *** Sukhdev has joined #openstack-infra | 17:34 | |
| mriedem | clarkb: i'll poke sdague | 17:34 |
| mriedem | sdague: poke! https://review.openstack.org/#/c/290832/ | 17:34 |
| clarkb | mriedem: I think you need to add the stable python jobs template in jenkins/jobs/projects.yaml too | 17:34 |
| clarkb | mriedem: let me find an example | 17:35 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 17:35 |
| *** sfinucan has quit IRC | 17:35 | |
| clarkb | mriedem: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n3818 the two bitrot lines there need to be added to designate | 17:35 |
| clarkb | mriedem: which is at https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n1015 | 17:36 |
| mriedem | clarkb: ah, ok | 17:36 |
| mriedem | thanks | 17:36 |
| *** pcaruana has joined #openstack-infra | 17:36 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Add periodic-kilo/liberty jobs to openstack/designate https://review.openstack.org/292617 | 17:37 |
| *** sc68cal has joined #openstack-infra | 17:39 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quotas support https://review.openstack.org/285110 | 17:40 |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quota ansible tests https://review.openstack.org/293072 | 17:40 |
| *** jlanoux has quit IRC | 17:41 | |
| *** mrmartin has quit IRC | 17:42 | |
| *** mriedem1 has joined #openstack-infra | 17:43 | |
| *** mriedem has quit IRC | 17:44 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pin puppetlabs-rabbitmq https://review.openstack.org/293074 | 17:45 |
| *** mriedem1 has quit IRC | 17:46 | |
| *** abregman has quit IRC | 17:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pin puppetlabs-rabbitmq https://review.openstack.org/293074 | 17:46 |
| *** rossella_s has quit IRC | 17:47 | |
| Zara | hi, we believe all storyboard email patches have landed (other instances send emails, and infra's mail config was merged last week), but emails still aren't arriving from storyboard.openstack.org . Anyone know what we might have missed? | 17:48 |
| *** sputnik13 has joined #openstack-infra | 17:48 | |
| *** derekh has quit IRC | 17:49 | |
| *** mrmartin has joined #openstack-infra | 17:51 | |
| krotscheck | Zara: It could be that the cloud which storyboard is running on does not permit outgoing mail. I seem to remember something like that. | 17:51 |
| krotscheck | fungi would know more. | 17:51 |
| krotscheck | Or clarkb | 17:52 |
| clarkb | krotscheck: it should be running in rax which does permit email (it is a big reason for why our control plane is in rax) | 17:52 |
| krotscheck | clarkb: Oh good. | 17:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test https://review.openstack.org/289445 | 17:52 |
| krotscheck | ON an unrelated note, I'm concerned about something the UX team is doing with their repo- they're proposing and committing binary files. | 17:52 |
| Zara | :0 | 17:52 |
| krotscheck | git... doesn't diff those. It just stores them as blobs. | 17:52 |
| clarkb | Zara: I can look at exim logs and see what there is to see | 17:52 |
| clarkb | Zara: but fungi and jeblair know far more about mail than I do | 17:53 |
| *** mriedem has joined #openstack-infra | 17:53 | |
| Zara | clarkb: brilliant, thank you. yeah, I know very little about it; annoying position to be in... | 17:53 |
| krotscheck | I anticipate that repository is going to balloooooooon. | 17:53 |
| clarkb | krotscheck: what kind of binary files? but generally yes that isn't considered the greatest of ideas | 17:53 |
| *** salv-orl_ has joined #openstack-infra | 17:54 | |
| *** ybathia has joined #openstack-infra | 17:54 | |
| krotscheck | clarkb: Turns out I may have spoken too soon. There's a PDF up for reveiw right now, but in the future they'll only accept svg | 17:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 17:55 |
| *** ybathia has left #openstack-infra | 17:55 | |
| clarkb | Zara: 2016-03-15 16:44:12 H=localhost (storyboard.openstack.org) [127.0.0.1] sender verify fail for <no-reply@storyboard.openstack.org>: Unknown user | 17:55 |
| clarkb | I think exim is saying it cna't use the no-reply user to send mail because it doesn't know who that user is? | 17:55 |
| clarkb | it is trying to send mail though | 17:55 |
| Zara | sounds like it, nice to know it's trying! :) | 17:56 |
| *** salv-orlando has quit IRC | 17:56 | |
| Zara | so I guess we want it to either know who that user is, or send from a different user... | 17:57 |
| openstackgerrit | Timur Sufiev proposed openstack-infra/project-config: Fix the location of devstack_gate_rc script at horizon job https://review.openstack.org/293077 | 17:58 |
| *** ihrachys has quit IRC | 17:58 | |
| fungi | clarkb: Zara: yes, i believe this came up before when i checked logs for SotK a month or more ago. exim is performing sender verification on locally-injected messages there. we should be able to configure it to not do that, or we could configure it to use a valid from address | 17:59 |
| clarkb | storyboard is a valid user so we could use storyboard@storyboard.openstack.org | 17:59 |
| *** bdemers has quit IRC | 17:59 | |
| anteaya | that sounds like an easy way forward | 17:59 |
| fungi | right, it mainly wants to know that there's somewhere it can send the non-delivery reports | 18:00 |
| Zara | if we send it to storyboard@storyboard.openstack.org, what happens if someone replies? | 18:00 |
| anteaya | does anyone recieve the mail sent to storyboard@storyboard.openstack.org? | 18:00 |
| Zara | s/to/from/ | 18:00 |
| *** Sukhdev has quit IRC | 18:00 | |
| anteaya | what Zara said | 18:00 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 18:00 |
| fungi | as an example, gerrit is sending as review@openstack.org | 18:01 |
| fungi | and setting the reply-to header to someone/somewhere it would make sense to reply | 18:01 |
| clarkb | exim is listening on port 25 so it should get delivered to storyboard | 18:01 |
| *** bdemers has joined #openstack-infra | 18:02 | |
| fungi | (in the case of gerrit review comments, it sets reply-to to the preferred e-mail address of the comment author) | 18:02 |
| fungi | not to imply that gerrit is an ideal model for dealing with e-mail notification, but it is at least _a_ model we currently handle and have for many years | 18:03 |
| *** _nadya_ has quit IRC | 18:03 | |
| *** dims_ has joined #openstack-infra | 18:03 | |
| clarkb | in storyboard's case it probably does make sense to deliver it to storyboard itself and have it do updates to tasks based on emails | 18:03 |
| *** dims has quit IRC | 18:03 | |
| clarkb | but that may be an eventual goal rather than an immediate worry | 18:04 |
| *** flongo has quit IRC | 18:04 | |
| pabelanger | ianw: centos7 dib is failing on https://rdoproject.org/repos/rdo-release.rpm | 18:04 |
| pabelanger | any ideas what we're getting from rdo? | 18:05 |
| persia | Having storyboard receive the mail for now, and ignore it is probably the easiest way forward. There have been discussions in the patches about having Storyboard parse email in the future. | 18:05 |
| fungi | yes, as a down-the-road feature opportunity, having the option for storyboard to receive and process replies nakes sense, so in anticipation of that having a bit bucket we can send replies into for now (storyboard@storyboard.openstack.org) seems time | 18:05 |
| fungi | er, seems fine | 18:05 |
| fungi | hah, what persia said more eloquently | 18:05 |
| jeblair | yeah, i wrote a router to do verp to so it can handle bounces | 18:05 |
| *** ddieterly is now known as ddieterly[away] | 18:05 | |
| jeblair | i think it should also do that; preferably sooner rather than later :) | 18:06 |
| Zara | :) okay, so first step, changing the sender and reply-to to storyboard@storyboard.openstack.org | 18:07 |
| *** tonytan4ever has quit IRC | 18:07 | |
| *** kzaitsev_mb has quit IRC | 18:07 | |
| pedroalvarez | Zara: I was clonning the repo to do that, are you doing it already? | 18:08 |
| jeblair | that should work but will cause bounces to be delivered to the storyboard user's mail spool (so that's probably best considered a temporary measure) | 18:08 |
| Zara | pedroalvarez: ah, you're ahead of me; I'd just found the right file | 18:08 |
| Zara | go for it | 18:08 |
| *** mrmartin has quit IRC | 18:10 | |
| *** lucasagomes is now known as lucas-dinner | 18:10 | |
| Zara | jeblair: okay, I'll bear it in mind. they need a lot of work in every direction... but it's a start! :) | 18:10 |
| fungi | right, we have until the rootfs fills up or runs out of inodes to get storyboard handling inbound e-mail and discarding or tracking bounces ;) | 18:10 |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack-infra/puppet-storyboard: Change default $sender_email_address https://review.openstack.org/293080 | 18:11 |
| thiagop | bumping my head against nodepool, image-build is getting stuck. How can I see logs or something to identify what happened? | 18:11 |
| *** terriyu has joined #openstack-infra | 18:11 | |
| anteaya | pedroalvarez: thank you | 18:12 |
| pabelanger | thiagop: you should see the output in image.log | 18:12 |
| thiagop | (I didn't run the service yet) | 18:12 |
| thiagop | pabelanger: in the same folder I ran image-build? | 18:12 |
| *** adrian_otto1 has joined #openstack-infra | 18:12 | |
| pedroalvarez | anteaya: :) | 18:12 |
| anteaya | there's that smile | 18:13 |
| pedroalvarez | tomorrow could be the day that storyboard sends emails :) | 18:13 |
| anteaya | such a wonderful smile, lights up a room it does | 18:13 |
| anteaya | :) | 18:13 |
| anteaya | pedroalvarez: yay tomorrow | 18:13 |
| pabelanger | thiagop: depends on how you installed it, and if logging.conf is setup | 18:13 |
| pabelanger | check /var/log/nodepool | 18:14 |
| *** sridhar_ram1 has joined #openstack-infra | 18:14 | |
| *** sridhar_ram has quit IRC | 18:14 | |
| thiagop | pabelanger: no logs on this folder. I installed it using puppet-openstackci | 18:14 |
| *** adrian_otto has quit IRC | 18:14 | |
| pabelanger | thiagop: check /etc/nodepool/logging.conf | 18:14 |
| pabelanger | it should point to /var/log/nodepool/image.log | 18:15 |
| pabelanger | or | 18:15 |
| pabelanger | /var/www/nodepool I think | 18:15 |
| *** jistr has quit IRC | 18:15 | |
| Zara | (:D once it works, the emails are off by default, so users can choose to trial them. because they're really annoyi-- er, they could do with being more configurable, but we'd like to know what to prioritize) | 18:15 |
| *** mrmartin has joined #openstack-infra | 18:16 | |
| AJaeger | bknudson: devstack should support openSUSE, there were two weeks ago 3 patches to fix some problems. Hope those merged now... | 18:16 |
| openstackgerrit | Merged openstack-infra/project-config: Skip magnum functional test jobs on docs changes https://review.openstack.org/277892 | 18:17 |
| thiagop | on builder-logging.conf it points to /var/log/nodepool/image/image.log, but this file isn't generated | 18:17 |
| anteaya | Zara: yup, thanks for the optional annoyance | 18:17 |
| *** _nadya_ has joined #openstack-infra | 18:17 | |
| thiagop | pabelanger: ^ | 18:17 |
| anteaya | and for jeblair for making it possible | 18:17 |
| sc68cal | AJaeger: ping me if those patches haven't | 18:17 |
| *** jordanP has quit IRC | 18:17 | |
| pabelanger | thiagop: is nodepool-builder configured to use it? | 18:17 |
| pabelanger | been a while since I used puppet-openstackci | 18:18 |
| pabelanger | thiagop: check debug.log too | 18:18 |
| thiagop | pabelanger: no debug.log or nodepool.log either. | 18:18 |
| pabelanger | Hmm | 18:18 |
| * thiagop feels like something is missing on the docs... | 18:18 | |
| pabelanger | Your the 2nd person this week to say logging wasn't working | 18:19 |
| pabelanger | I wonder if some step is missing | 18:19 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate https://review.openstack.org/293023 | 18:19 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 18:19 |
| *** infra-red has joined #openstack-infra | 18:19 | |
| thiagop | pabelanger: http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool | 18:19 |
| thiagop | I'm on the image-build step of this^ | 18:20 |
| pabelanger | Oh | 18:20 |
| *** dizquierdo has quit IRC | 18:20 | |
| pabelanger | that's because you are not using service nodepool-builder start | 18:20 |
| thiagop | I found it strange that we run this thing without nodepool service is actually running | 18:20 |
| pabelanger | no, my mistake | 18:20 |
| pabelanger | the server should be running | 18:20 |
| *** flongo has joined #openstack-infra | 18:21 | |
| pabelanger | ps -ef | grep nodepool | 18:21 |
| *** flongo has quit IRC | 18:21 | |
| *** terriyu has quit IRC | 18:22 | |
| thiagop | pabelanger: no mention of nodepool-builder, but the service is here (stopped) | 18:22 |
| thiagop | no mention *on the docs* | 18:22 |
| AJaeger | sc68cal: let me check and get back to you... | 18:22 |
| thiagop | pabelanger: nothing special on ps: http://paste.openstack.org/show/490575/ | 18:23 |
| openstackgerrit | Merged openstack-infra/project-config: Add periodic-kilo/liberty jobs to openstack/designate https://review.openstack.org/292617 | 18:23 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental job to test xstatic packaging https://review.openstack.org/289125 | 18:23 |
| *** _nadya_ has quit IRC | 18:23 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Refactoring https://review.openstack.org/292912 | 18:24 |
| *** vgridnev has joined #openstack-infra | 18:24 | |
| pabelanger | thiagop: Ya, looks like nodepool service is not running | 18:25 |
| pabelanger | thiagop: sudo service nodepool status | 18:25 |
| pabelanger | thiagop: sudo service nodepool start | 18:25 |
| pabelanger | then, check ps again, ensure -l /etc/nodepool/logging.conf is passed | 18:25 |
| *** sridhar_ram1 is now known as sridhar_ram | 18:26 | |
| pabelanger | then, try image-build again | 18:26 |
| *** flepied has quit IRC | 18:26 | |
| pabelanger | thiagop: you'll see something like the following when working: http://nodepool.openstack.org/dib.devstack-centos7.log | 18:26 |
| fungi | the instructions should probably say to manually start the service. our puppet modules generally don't ensure services are started because this can have the nasty effect of puppet automatically restarting the service when you've intentionally stopped it, or restarting it over and over when you've done something that makes it crash | 18:26 |
| pabelanger | Ya, that could be the issue | 18:27 |
| *** daemontool has quit IRC | 18:27 | |
| *** baoli_ has quit IRC | 18:27 | |
| *** IlyaG has quit IRC | 18:27 | |
| *** piet has quit IRC | 18:28 | |
| *** terriyu has joined #openstack-infra | 18:28 | |
| *** fhubik_real has quit IRC | 18:28 | |
| *** e0ne has joined #openstack-infra | 18:28 | |
| *** piet has joined #openstack-infra | 18:28 | |
| *** mrmartin has quit IRC | 18:29 | |
| *** dims_ has quit IRC | 18:29 | |
| *** esikachev has joined #openstack-infra | 18:29 | |
| thiagop | fungi: now I have debug.log, but no image.log yet | 18:30 |
| *** tonytan4ever has joined #openstack-infra | 18:30 | |
| thiagop | pabelanger: ^ | 18:30 |
| thiagop | debug log keeps repeating this: http://paste.openstack.org/show/490576/ | 18:31 |
| thiagop | should I start the nodepool-builder too? | 18:31 |
| openstackgerrit | Shiv Haris proposed openstack-infra/project-config: Adding branch creation priv to group networking-brocade-release https://review.openstack.org/292635 | 18:33 |
| thiagop | this is the debug of "nodepool image-build" | 18:33 |
| thiagop | http://paste.openstack.org/show/490577/ | 18:33 |
| jeblair | thiagop: yes, the builder service is responsible for building and uploading images | 18:33 |
| *** [1]Thelo has joined #openstack-infra | 18:33 | |
| jeblair | thiagop: that paste indicates there is a job queued for the builder, and it will run as soon as you start the builder | 18:34 |
| *** sdake_ has quit IRC | 18:34 | |
| thiagop | jeblair: so something is missing in that doc... | 18:34 |
| jeblair | thiagop: yep | 18:34 |
| *** degorenko is now known as _degorenko|afk | 18:34 | |
| thiagop | jeblair: I'll send a patch as soon as I discover which file to update | 18:35 |
| jeblair | thanks! | 18:35 |
| fungi | thiagop: which document were you following? i'm happy to point you to the source for whatever it is | 18:35 |
| thiagop | http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool | 18:35 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate https://review.openstack.org/293023 | 18:35 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 18:35 |
| *** Thelo has quit IRC | 18:35 | |
| *** [1]Thelo is now known as Thelo | 18:35 | |
| *** esikachev has quit IRC | 18:36 | |
| *** vgridnev has quit IRC | 18:37 | |
| *** vgridnev has joined #openstack-infra | 18:39 | |
| AJaeger | sc68cal, bknudson: The suse changes for devstack have all merged. If anything is not ok, please ask toabctl who's more active in this area than me. | 18:39 |
| fungi | thiagop: the source rts file for that document is http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/doc/source/third_party_ci.rst | 18:40 |
| sc68cal | AJaeger: ack. thanks | 18:40 |
| fungi | thiagop: changes should be submitted through gerrit just like for the rest of our tooling | 18:40 |
| thiagop | fungi: thanks! will do next | 18:40 |
| anteaya | pleia2: when did we stop paginating git.o.o, do you know? | 18:40 |
| fungi | er, s/rts/rst/ | 18:40 |
| fungi | anteaya: there was a recent change i saw fly by to up the page length again to something we're unlikely to hit for a very long time | 18:40 |
| anteaya | it makes searching faster, I do find but I was caught by surprise by the change | 18:40 |
| anteaya | fungi: ah okay, so intentional | 18:41 |
| anteaya | thanks | 18:41 |
| *** dalgaaf has joined #openstack-infra | 18:42 | |
| anteaya | well the change wasn't to puppet-cgit: http://git.openstack.org/cgit/openstack-infra/puppet-cgit/log/ | 18:42 |
| anteaya | would it have been in system-config? | 18:42 |
| *** terriyu has quit IRC | 18:42 | |
| fungi | anteaya: yep, https://review.openstack.org/288745 | 18:43 |
| fungi | so that changed about 5-6 days ago | 18:43 |
| anteaya | cool, had just found that myself | 18:44 |
| anteaya | always so slow compared to you :) | 18:44 |
| fungi | though that was 1500 and we're already well over 1000, so maybe "a very long time" is not so long, depending on how you measure | 18:44 |
| anteaya | measureing in fuel units? | 18:44 |
| anteaya | might have to up it by the end of the month | 18:44 |
| fungi | hah | 18:44 |
| *** BobBall has quit IRC | 18:44 | |
| fungi | how many xstatics does it take to equal one fuel? | 18:45 |
| anteaya | I haven't done the converstaion | 18:45 |
| anteaya | my eyes blur when I see either word | 18:45 |
| fungi | i think the rates fluctuate daily | 18:45 |
| anteaya | like the stock market | 18:45 |
| fungi | better to invest in mordred's spirit animal fund | 18:45 |
| fungi | it seems to pay hefty dividends on a consistent schedule | 18:46 |
| anteaya | it does at that | 18:46 |
| anteaya | never seems to run low | 18:46 |
| * sc68cal chuckles | 18:48 | |
| *** asettle has joined #openstack-infra | 18:48 | |
| AJaeger | infra-root, this morning yolanda approved new repos http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/ and http://git.openstack.org/cgit/openstack/tripleo-ui/ - but those are not imported in git.o.o yet "Repository seems to be empty". Also they are not setup in gerrit | 18:49 |
| AJaeger | could somebody look at these, please? | 18:49 |
| *** _amrith_ is now known as amrith | 18:49 | |
| fungi | AJaeger: yes, yolanda was previously investigating why puppet doesn't seem to have successfully updated review.o.o since the 10th but we all got so wrapped up in the zuul config issue i think we forgot to follow up | 18:49 |
| fungi | as such, our image updates for today are probably all shot | 18:50 |
| *** e0ne_ has joined #openstack-infra | 18:50 | |
| *** ddieterly[away] is now known as ddieterly | 18:51 | |
| *** e0ne has quit IRC | 18:51 | |
| jeblair | fungi, pleia2: we're going to have an election! i think that's wonderful and exciting! i hope you both win! | 18:51 |
| nibalizer | hrm syslog doesn't have any puppet runs on review.o.o | 18:51 |
| openstackgerrit | Thiago Paiva Brito proposed openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst https://review.openstack.org/293087 | 18:51 |
| thiagop | fungi jeblair ^ | 18:51 |
| fungi | jeblair: i need a volunteer to run the electorate generation script for me on the day before the election and get the results to tristanC and tonyb | 18:52 |
| fungi | jeblair: are you free to do that? | 18:52 |
| jeblair | fungi: what day is that? | 18:52 |
| fungi | i'll make sure the script is retested and working as intended in the next day or so | 18:52 |
| tonyb | jeblair: March 17th (UTC) | 18:52 |
| fungi | though it was working fine for me as of a couple weeks ago | 18:52 |
| *** PsionTheory has joined #openstack-infra | 18:52 | |
| fungi | so i don't expect any surprises | 18:53 |
| *** asettle has quit IRC | 18:53 | |
| *** electrofelix has quit IRC | 18:53 | |
| openstackgerrit | Rob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack https://review.openstack.org/293090 | 18:53 |
| *** oanson has joined #openstack-infra | 18:54 | |
| *** vgridnev has quit IRC | 18:55 | |
| openstackgerrit | Thiago Paiva Brito proposed openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst https://review.openstack.org/293087 | 18:55 |
| *** adrian_otto1 has quit IRC | 18:56 | |
| pabelanger | fungi: does new channels for accessbot cause the bot to drop from IRC? | 18:57 |
| pabelanger | or is that meetbot? | 18:57 |
| fungi | meetbot | 18:57 |
| fungi | changes to the meetbot channel list will cause the "openstack" meetbot to leave and rejoin due to restarting, and so lose its state on running meetings | 18:57 |
| AJaeger | pabelanger: system-config IRC changes are the problem ;) | 18:57 |
| AJaeger | pabelanger: project-config is fine | 18:58 |
| *** adrian_otto has joined #openstack-infra | 18:58 | |
| AJaeger | pabelanger: that's my rule of thumb ;) | 18:58 |
| jeblair | fungi: thursday pacific time is not great for me actually; i'm traveling/meeting. of course, i could do it just after midnight utc (wednesday evening pacific) but perhaps it would be better to find someone else who is more available thursday in case there are issues... | 18:58 |
| pabelanger | AJaeger: okay, I'll wait for osic change until later tonight | 18:58 |
| fungi | jeblair: sure, i'll see who's available with gerrit admin rights | 18:58 |
| fungi | since we haven't yet switched to a build of gerrit with zaro's backport which would make this runnable by everyone | 18:58 |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| jeblair | fungi: k. i'm on board if you can't find someone with better availability. | 19:00 |
| *** adrian_otto has quit IRC | 19:00 | |
| *** flepied has joined #openstack-infra | 19:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Move layout job to bare-precise https://review.openstack.org/293015 | 19:01 |
| openstackgerrit | Merged openstack-infra/project-config: Add CentOS & Fedora dashboards to grafana https://review.openstack.org/292123 | 19:02 |
| yolanda | fungi i wanted to check with mordred, about puppet not running on review. I was worried about some strange situation with having 2 sites called review.openstack.org | 19:02 |
| *** flwang1 has quit IRC | 19:03 | |
| ianw | pabelanger: i think that was the problem yesterday. the timestamp suggests we haven't tried building again... | 19:03 |
| *** kushal has quit IRC | 19:03 | |
| anteaya | yolanda: oh good point | 19:03 |
| *** bnemec has quit IRC | 19:03 | |
| *** vgridnev has joined #openstack-infra | 19:04 | |
| pabelanger | ianw: okay, I need to look into system-config and see what RDO we use | 19:04 |
| ianw | pabelanger: the reason we need it is for ovs | 19:06 |
| openstackgerrit | Merged openstack-infra/system-config: Make sure wheel-mirror works on centos7 https://review.openstack.org/290146 | 19:06 |
| pabelanger | ianw: Ah, thank makes sense | 19:07 |
| *** sdake has joined #openstack-infra | 19:07 | |
| ianw | pabelanger: the bigger question is, why didn't it retry the build today. i haven't read scrollback but i guess that was not expected | 19:07 |
| *** pvaneck has quit IRC | 19:10 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Refactoring https://review.openstack.org/292912 | 19:10 |
| *** dims has joined #openstack-infra | 19:12 | |
| *** bnemec has joined #openstack-infra | 19:13 | |
| *** krotscheck is now known as kro_death_by_sin | 19:14 | |
| *** fverbs has quit IRC | 19:14 | |
| *** kro_death_by_sin is now known as kro_sinus_hell | 19:14 | |
| *** bnemec has quit IRC | 19:16 | |
| *** dprince has quit IRC | 19:18 | |
| *** bnemec has joined #openstack-infra | 19:18 | |
| *** dims has quit IRC | 19:18 | |
| *** e0ne_ has quit IRC | 19:19 | |
| *** xyang1 has joined #openstack-infra | 19:19 | |
| *** ybathia has joined #openstack-infra | 19:21 | |
| *** sdake_ has joined #openstack-infra | 19:21 | |
| *** sdake has quit IRC | 19:23 | |
| *** dims has joined #openstack-infra | 19:26 | |
| *** sridhar_ram has quit IRC | 19:28 | |
| pabelanger | ianw: Ya, I am not sure. Would need to check the logs myself too. I can do that after meetings | 19:29 |
| ddieterly | stack.sh is hanging at this point | 19:30 |
| ddieterly | sudo grep -q "^#includedir.*/etc/sudoers.d" /etc/sudoers || | 19:30 |
| ddieterly | echo "#includedir /etc/sudoers.d" | sudo tee -a /etc/sudoers | 19:30 |
| ddieterly | any ideas why? | 19:30 |
| ddieterly | i'm following the instructions at https://github.com/openstack-infra/devstack-gate | 19:31 |
| *** dprince has joined #openstack-infra | 19:31 | |
| *** ociuhandu has quit IRC | 19:32 | |
| *** ybathia has quit IRC | 19:33 | |
| *** fawadkhaliq has quit IRC | 19:35 | |
| *** PsionTheory has quit IRC | 19:36 | |
| *** aysyd_ has joined #openstack-infra | 19:37 | |
| *** aysyd has quit IRC | 19:37 | |
| *** aysyd_ is now known as aysyd | 19:37 | |
| *** hashar has joined #openstack-infra | 19:39 | |
| *** nwkarsten has quit IRC | 19:40 | |
| ianw | ddieterly: which bit is hanging? might be useful to put logs in paste.openstack.org . also, more a #openstack-qa type question | 19:41 |
| ddieterly | ianw seems like every sudo command is hanging in stack.sh | 19:42 |
| ddieterly | ianw which log file is it? i can't seem to find anything useful in the log | 19:42 |
| *** cody-somerville has quit IRC | 19:42 | |
| *** tsg has joined #openstack-infra | 19:43 | |
| *** cody-somerville has joined #openstack-infra | 19:43 | |
| *** salv-orl_ has quit IRC | 19:43 | |
| ianw | ddieterly: have you set NOPASSWD for your user? | 19:43 |
| ddieterly | mmm... i don't think so | 19:44 |
| ddieterly | that sounds like the problem | 19:44 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate https://review.openstack.org/293108 | 19:44 |
| ddieterly | oh, yes i did | 19:44 |
| *** _nadya_ has joined #openstack-infra | 19:45 | |
| *** dims has quit IRC | 19:45 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 19:46 |
| *** dprince has quit IRC | 19:48 | |
| *** abregman has joined #openstack-infra | 19:48 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 19:50 |
| *** sdake_ has quit IRC | 19:51 | |
| *** pvaneck has joined #openstack-infra | 19:52 | |
| asselin | thiagop, jeblair fungi pabelanger fyi those instructions were accurate as of nodepool 0.2.0. They need to be updated to the new version of nodepool, or 3rd party ci requested to pin to 0.2.0 (latest tag) | 19:52 |
| *** ociuhandu has joined #openstack-infra | 19:53 | |
| *** sridhar_ram has joined #openstack-infra | 19:54 | |
| pabelanger | asselin: should we add a nice fat warning to the page? | 19:56 |
| ddieterly | looks like the stack user also needs the NOPASSWD setting | 19:56 |
| *** rockyg has joined #openstack-infra | 19:58 | |
| *** MarkAtwood has joined #openstack-infra | 19:58 | |
| anteaya | david-lyle: my goodness, you are stilling getting accolades | 19:59 |
| anteaya | david-lyle: can we make the david lyle going away parade into an annual event? | 19:59 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Pin nodepool to latest tag 0.2.0 for 3rd party CI https://review.openstack.org/293112 | 19:59 |
| david-lyle | anteaya: stockholm syndrome is real | 20:00 |
| asselin | pabelanger, ^^ I can also add to the docs saying this is assuming version 0.2.0 | 20:00 |
| anteaya | david-lyle: ha ha ha | 20:00 |
| *** annegentle has joined #openstack-infra | 20:01 | |
| * craige reminds -infra folk that he needs more eyes on his dev phabricator instance: https://os01.mcwhirter.com.au/ - it's very close to ready and I need more eyes identifying issues. Thanks! | 20:02 | |
| anteaya | piet: ^^ | 20:02 |
| piet | Yep | 20:03 |
| craige | and txt :-) | 20:03 |
| craige | ugh ttx | 20:03 |
| pabelanger | asselin: might be worth it | 20:03 |
| *** tonytan4ever has quit IRC | 20:04 | |
| piet | craige Does it include Pholio? | 20:04 |
| *** ig0r_ has joined #openstack-infra | 20:04 | |
| *** tonytan4ever has joined #openstack-infra | 20:04 | |
| craige | IIRC yes (airport) piet . if not, I'm happy to add it. | 20:05 |
| *** ig0r__ has quit IRC | 20:06 | |
| piet | craige The OpenStack UX team is looking to move away from Invision (closed source mock review tool) to Pholio | 20:06 |
| *** erikwilson has joined #openstack-infra | 20:06 | |
| *** vgridnev has quit IRC | 20:07 | |
| piet | craige We are getting a lot of complaints about Invision being too slow, so would like to move sooner than later | 20:08 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:08 |
| *** IlyaG has joined #openstack-infra | 20:09 | |
| thiagop | I think DIB is breaking due to this commit: https://github.com/openstack-infra/project-config/commit/b1a806767ea4e7b8fd280d8189f4c0dfd6859845 | 20:10 |
| *** vgridnev has joined #openstack-infra | 20:10 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 20:10 |
| *** dims has joined #openstack-infra | 20:11 | |
| *** sdake has joined #openstack-infra | 20:11 | |
| thiagop | os_designate isn't on git.openstack.org, it has an 'upstream' directive that should have been considered on the openstack-repos element | 20:11 |
| *** erikwilson has quit IRC | 20:11 | |
| *** adrian_otto has joined #openstack-infra | 20:12 | |
| *** abregman is now known as abregman|afk | 20:12 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Clarify nodepool image build instructions post 0.2.0 https://review.openstack.org/293122 | 20:12 |
| asselin | pabelanger, thiagop ^^ | 20:12 |
| *** vgridnev has quit IRC | 20:13 | |
| dmsimard | pabelanger: turns out there *is* an equivalent to the puppet metadata.json for galaxy | 20:13 |
| dmsimard | meta/main.yml | 20:13 |
| ianw | pabelanger / fungi : can someone check the nodepool builder logs and see why it seems to have stopped? it would also be nice to restart it, because it hasn't picked up centos-minimal jobs | 20:13 |
| thiagop | asselin: the line ahead explains how to start nodepool, not nodepool-builder | 20:13 |
| *** esker has joined #openstack-infra | 20:14 | |
| thiagop | asselin: am I ignoring something? | 20:14 |
| asselin | thiagop, I haven't used nodepool after tag 0.2.0 yet so I cannot update the instructions. You can switch back to tag 0.2.0 or look at the nodepool documentation whch is hoepfully up-to-date | 20:15 |
| asselin | thiagop, http://docs.openstack.org/infra/nodepool/ | 20:15 |
| smarcet | fungi: not sure what happens here https://review.openstack.org/#/c/292912/ | 20:15 |
| smarcet | fungi: but its failling, i tried same commands locally and works just fine | 20:16 |
| asselin | thiagop, if you want to use the 0.2.0 tag: https://review.openstack.org/293112 | 20:16 |
| *** kgiusti has left #openstack-infra | 20:16 | |
| thiagop | asselin: the nodepool docs were surprisingly not helpful for me... :/ | 20:18 |
| mgagne | what's nodepool-builder? o_O | 20:19 |
| pabelanger | dmsimard: Oh, I knew that | 20:19 |
| pabelanger | dmsimard: just didn't register that's what you were looking for | 20:19 |
| pabelanger | ianw: fungi: I can take a peak | 20:20 |
| dmsimard | pabelanger: no worries :p | 20:20 |
| fungi | smarcet: could it be something to do with the default php pear search path? | 20:20 |
| smarcet | not sure | 20:21 |
| fungi | smarcet: my php hacking days are so far behind me that it's a somewhat opaque topic for me these days | 20:21 |
| fungi | smarcet: but i'll stare at it a little more and see if i come up with anything | 20:21 |
| smarcet | i mean didnt change anything related to that | 20:21 |
| pabelanger | fungi: ianw: last image to build was ubuntu-trusty at 2016-03-14 15:45:45,436 but was a failure | 20:22 |
| *** tonytan4ever has quit IRC | 20:22 | |
| *** tonytan4ever has joined #openstack-infra | 20:23 | |
| pabelanger | sorry, it retried at 2016-03-14 16:36:08,054 and was successful | 20:23 |
| fungi | pabelanger: ianw: as i mentioned earlier, puppet not applying on review.o.o means several new repos broken on git.o.o means all nodepool image builds are going to bomb today | 20:23 |
| *** kzaitsev_mb has joined #openstack-infra | 20:23 | |
| *** rfolco has quit IRC | 20:23 | |
| *** matrohon has joined #openstack-infra | 20:23 | |
| *** rfolco has joined #openstack-infra | 20:24 | |
| ianw | fungi: ok, well what's a bit more concerning is they don't seem to have run at all, rather than failed. or at least logs on nodepool.openstack.org don't seem updated | 20:24 |
| nibalizer | fungi: review.openstack.org doesn't exist in ansible all --list-hosts | 20:25 |
| ianw | unless, possibly, dib is stuck on a git hang | 20:25 |
| thiagop | mgagne: appears to be a service to listen to image-build requests and pass it/monitors DIB execution | 20:25 |
| nibalizer | i wonder if its going to uuids since we have two review.o.o ? | 20:25 |
| pabelanger | fungi: ianw: Ya, I don't see any image builds for the 15th | 20:25 |
| thiagop | mgagne: but I may be wrong, first day digging into it | 20:25 |
| pabelanger | don't see any errors right now | 20:25 |
| thiagop | :) | 20:25 |
| *** nwkarsten has joined #openstack-infra | 20:25 | |
| *** nwkarsten has quit IRC | 20:25 | |
| *** nwkarsten has joined #openstack-infra | 20:26 | |
| fungi | nibalizer: yolanda theorized that her by-uuid blacklist for the replacement review.o.o is somehow responsible | 20:26 |
| fungi | nibalizer: but agreed if the inventory switches to only uuid for both of those, then ansible/puppet may get interesting? | 20:27 |
| ianw | thiagop / mgagne : nodepool-builder is what builds the images. it was separated into a separate process from the main nodepool daemon a little while ago. | 20:27 |
| mgagne | ianw: still optional? | 20:27 |
| *** roxanagh_ has joined #openstack-infra | 20:28 | |
| *** oanson has quit IRC | 20:28 | |
| *** sridhar_ram1 has joined #openstack-infra | 20:29 | |
| *** bswartz has quit IRC | 20:29 | |
| nibalizer | how is the by-uuid blacklist implemented? | 20:30 |
| *** dimtruck is now known as zz_dimtruck | 20:30 | |
| pabelanger | fungi: which syntax should I be using to send a nodepool command? I believe you use sudo over su nodpool? | 20:30 |
| *** piet has quit IRC | 20:30 | |
| *** sridhar_ram has quit IRC | 20:30 | |
| *** bswartz has joined #openstack-infra | 20:30 | |
| *** piet has joined #openstack-infra | 20:30 | |
| *** keedya has joined #openstack-infra | 20:31 | |
| *** flwang1 has joined #openstack-infra | 20:31 | |
| pabelanger | ianw: fungi: I think they are queued to be built, about to confirm | 20:31 |
| ianw | mgagne: having it as a separate daemon is still optional | 20:32 |
| *** keedya has quit IRC | 20:32 | |
| ianw | mgagne: see --no-builder option | 20:32 |
| smarcet | fungi: no worries, i was able to reproduce it locally | 20:32 |
| smarcet | workiung on it | 20:32 |
| *** roxanagh_ has quit IRC | 20:32 | |
| fungi | smarcet: oh, good, because i'm also testing several other things and trying to follow the tc meeting right now, so my attention was a little divided. sorry! | 20:33 |
| fungi | pabelanger: sudo -H -u nodepool nodepool list | 20:33 |
| fungi | pabelanger: et cetera | 20:33 |
| mgagne | ianw: thanks for the info. the doc is not very explicit about it tbh | 20:34 |
| *** regXboi has quit IRC | 20:34 | |
| *** fitoduarte has joined #openstack-infra | 20:34 | |
| *** gordc has quit IRC | 20:34 | |
| fungi | pabelanger: it's more crucial for dib operations since those create files on the filesystem in a predictable location, so not doing that as root is more sanitary, and also doing it as root leaves files that the nodepool user will later fail to be able to remove/replace | 20:34 |
| *** esikachev has joined #openstack-infra | 20:34 | |
| *** oanson has joined #openstack-infra | 20:34 | |
| pabelanger | fungi: roger | 20:35 |
| pabelanger | | 176408 | centos-7 | /opt/nodepool_dib/centos-7-1458051067 | 1458051067 | building | 00:06:23:26 | | 20:35 |
| pabelanger | fungi: ianw ^ | 20:35 |
| pabelanger | looks like dib is queued to build | 20:35 |
| pabelanger | but nodepool-builder didn't do it | 20:35 |
| thiagop | pabelanger: | 20:35 |
| thiagop | that was me | 20:35 |
| thiagop | I started the builder and it worked | 20:35 |
| *** MarkAtwood has quit IRC | 20:36 | |
| ianw | pabelanger: yeah, we know about that one, becaues the builder hasn't reloaded and thus hasn't registered to consume the job | 20:36 |
| thiagop | I followed the process by tail -f on /var/log/nodepool/images/images.log | 20:36 |
| thiagop | image.log* | 20:36 |
| *** maishsk has joined #openstack-infra | 20:36 | |
| *** e0ne has joined #openstack-infra | 20:37 | |
| *** flwang1 has quit IRC | 20:37 | |
| ianw | pabelanger: does ps show any stuck dib processes? | 20:37 |
| *** salv-orlando has joined #openstack-infra | 20:37 | |
| ianw | although that doesn't make sense, we should still get logs | 20:37 |
| *** rockyg has quit IRC | 20:37 | |
| *** esker has quit IRC | 20:38 | |
| *** rockyg has joined #openstack-infra | 20:38 | |
| thiagop | the builder has a log of its own too | 20:38 |
| pabelanger | ianw: Ya, so besides that, I don't see a reason why new dibs were not created today. I only see centos7 being added to the queue at 14:00 UTC | 20:38 |
| *** esker has joined #openstack-infra | 20:38 | |
| pabelanger | other dibs were skipped | 20:38 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add a cluster health indicator to the graph output https://review.openstack.org/290832 | 20:39 |
| *** esikachev has quit IRC | 20:39 | |
| pabelanger | thiagop: good to know you got it working | 20:39 |
| thiagop | pabelanger: I'm now fixing the openstack-repos element and testing | 20:39 |
| thiagop | pabelanger: appears that a commit yesterday had an out-of-the-box approach | 20:40 |
| *** baoli has joined #openstack-infra | 20:41 | |
| pabelanger | ianw: so, we could kick off a build manually, but like fungi said, going to fail to clone repos. Guess we hold off until gerrit is fixed | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Remove gate-tempest-dsvm-full-ceph job https://review.openstack.org/293130 | 20:41 |
| mriedem | sdague: dansmith: jbernard_: flaper87: smcginnis_away: ^ | 20:41 |
| *** fawadkhaliq has joined #openstack-infra | 20:42 | |
| *** IlyaG has quit IRC | 20:43 | |
| anteaya | mriedem: you do realize that the -nv at the end of the job means non-voting? | 20:43 |
| mriedem | anteaya: i do | 20:43 |
| anteaya | mriedem: so you would make a copy of the -nv job and remove the -nv bit? | 20:43 |
| AJaeger | mriedem: see also https://review.openstack.org/273326 | 20:43 |
| *** MarkAtwood has joined #openstack-infra | 20:44 | |
| *** zz_dimtruck is now known as dimtruck | 20:44 | |
| mriedem | AJaeger: we aren't making it voting yet | 20:44 |
| mriedem | the plugin job that is | 20:44 |
| AJaeger | mriedem: ah... | 20:44 |
| mriedem | jbernard_ said this morning that was waiting on some mirror updates | 20:45 |
| jbernard_ | mriedem: yeah, as soon as clarkb gives me the ok, ill upde the plugin to fetch from the mirror | 20:45 |
| *** tongli has quit IRC | 20:46 | |
| clarkb | I mean its been low on my priority list | 20:46 |
| clarkb | (been trying to get things that affect the release done like logstash 2.0 and new clouds and so on) | 20:47 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Remove gate-tempest-dsvm-full-ceph job https://review.openstack.org/293130 | 20:47 |
| smarcet | fungi: i think that problem its the latest composer.phar version | 20:47 |
| smarcet | with older version does not happens | 20:47 |
| clarkb | I can start poking at fixing the puppet module though | 20:47 |
| mriedem | clarkb: re: ceph, we understand, was just pointing out why we're not making the plugin job voting yet | 20:47 |
| mriedem | i really just don't want to run redundant jobs on 3 projects when they are all nonvoting | 20:48 |
| *** IlyaG has joined #openstack-infra | 20:49 | |
| *** ihrachys has joined #openstack-infra | 20:50 | |
| openstackgerrit | lifeless proposed openstack/requirements: Ignore more build temp dirs https://review.openstack.org/251604 | 20:50 |
| *** sdake has quit IRC | 20:50 | |
| *** vgridnev has joined #openstack-infra | 20:51 | |
| clarkb | wow new puppet is typed? | 20:51 |
| zigo | In my tempest tests, I'm running Tempest with 2 testr threads. Am I right that this may cause some problems? | 20:52 |
| zigo | Should I remove the --parallel thing? | 20:52 |
| *** SamYaple is now known as NotSamYaple | 20:52 | |
| nibalizer | arst | 20:52 |
| clarkb | zigo: we run it with 4 threads on most of our jobs | 20:53 |
| clarkb | zigo: the value we use is numprocs/2 | 20:53 |
| zigo | clarkb: How much RAM do you have available? | 20:53 |
| *** sdake has joined #openstack-infra | 20:53 | |
| clarkb | zigo: this allows other processes like openstack itself and mysql to run on the other half of the cus | 20:53 |
| clarkb | zigo: 8GB | 20:53 |
| *** NotSamYaple is now known as SamYaple | 20:53 | |
| zigo | I noticed, because I'm running it on a 12 core machines, and I had 24 threads running, which failed some tests ... :) | 20:53 |
| *** _nadya_ has quit IRC | 20:53 | |
| AJaeger | could some project-config cores, please review a rework the of the api-guide jobs as templates so that we can use it for more projects? Change is https://review.openstack.org/#/c/292135 - and needed by https://review.openstack.org/#/c/285519. And https://review.openstack.org/#/c/292888 is part of the series | 20:53 |
| *** ybathia has joined #openstack-infra | 20:55 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Refactoring https://review.openstack.org/292912 | 20:55 |
| *** keedya has joined #openstack-infra | 20:56 | |
| *** jpr has quit IRC | 20:57 | |
| *** moravec has left #openstack-infra | 20:58 | |
| *** baoli has quit IRC | 20:58 | |
| *** flwang1 has joined #openstack-infra | 20:58 | |
| *** ddieterly is now known as ddieterly[away] | 20:59 | |
| *** baoli has joined #openstack-infra | 20:59 | |
| *** sridhar_ram1 has quit IRC | 21:01 | |
| *** moravec has joined #openstack-infra | 21:01 | |
| *** _ari_ has quit IRC | 21:01 | |
| *** annegentle has quit IRC | 21:02 | |
| *** sdake_ has joined #openstack-infra | 21:02 | |
| *** gordc has joined #openstack-infra | 21:03 | |
| *** thorst_ has quit IRC | 21:03 | |
| *** angdraug has joined #openstack-infra | 21:03 | |
| *** f1ller is now known as filler | 21:03 | |
| *** infra-red has quit IRC | 21:04 | |
| openstackgerrit | lifeless proposed openstack/requirements: Note that MySQL-python is more than just GPL. https://review.openstack.org/293138 | 21:04 |
| *** sdake has quit IRC | 21:05 | |
| *** MarkAtwood has quit IRC | 21:05 | |
| *** keedya has quit IRC | 21:05 | |
| *** rfolco has quit IRC | 21:06 | |
| *** aysyd has quit IRC | 21:06 | |
| *** smarcet has quit IRC | 21:07 | |
| bswartz | anyone know where to find logs for jobs that run in release pipe? | 21:07 |
| *** bdemers has quit IRC | 21:08 | |
| AJaeger | logs.openstack.org | 21:09 |
| *** thorst_ has joined #openstack-infra | 21:09 | |
| AJaeger | And use the SHA - firsttwochars/complete-sha | 21:09 |
| *** sdague has quit IRC | 21:09 | |
| *** eharney has quit IRC | 21:09 | |
| bswartz | AJaeger: http://logs.openstack.org/8a/8a7e5d27cbc6e5345a1eb86789482defbfbdb934/ | 21:09 |
| bswartz | I see only "post" here | 21:09 |
| bswartz | not release | 21:09 |
| AJaeger | bswartz: what's the tag and project? | 21:09 |
| bswartz | http://git.openstack.org/cgit/openstack/manila-image-elements/tag/?h=1.1.0 | 21:10 |
| *** shardy has quit IRC | 21:10 | |
| bswartz | AJaeger: ^ | 21:10 |
| AJaeger | bswartz: run "git showref 1.1.0" | 21:10 |
| AJaeger | to get the sha of the release | 21:10 |
| openstackgerrit | Merged openstack-infra/openstackid: Refactoring https://review.openstack.org/292912 | 21:10 |
| bswartz | ty that worked | 21:11 |
| AJaeger | bswartz: looking at your URL, it's d09924e72e292d8ee321e7e9a8b189885f2e5218 | 21:11 |
| bswartz | yeah | 21:11 |
| bswartz | http://logs.openstack.org/d0/d09924e72e292d8ee321e7e9a8b189885f2e5218/release/gate-manila-publishimage-generic/7186289/ | 21:11 |
| bswartz | now to find why it failed.... | 21:11 |
| *** keedya has joined #openstack-infra | 21:11 | |
| *** baoli has quit IRC | 21:13 | |
| jeblair | jesusaur: let's drop gear debug logs for now | 21:13 |
| *** thorst_ has quit IRC | 21:13 | |
| *** baoli has joined #openstack-infra | 21:13 | |
| bswartz | btw I never knew that tags get their own commit hash | 21:14 |
| * bswartz learned something new about git | 21:14 | |
| *** maishsk has quit IRC | 21:15 | |
| clarkb | mriedem: jbernard_ http://download.ceph.com/debian-hammer/ is what we want mirrored? its a bit annoying that its a mirror per ceph release x distro release | 21:15 |
| *** maishsk_ has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | but I guess that allows ceph to get stuff out to users in a very controlled fashion | 21:15 |
| *** thorst_ has joined #openstack-infra | 21:15 | |
| *** jtomasek has quit IRC | 21:16 | |
| lifeless | AJaeger: are you waiting on me for anything? | 21:17 |
| *** Guest93664 is now known as med_ | 21:17 | |
| *** med_ has joined #openstack-infra | 21:17 | |
| *** jtomasek has joined #openstack-infra | 21:18 | |
| AJaeger | lifeless: no, I'm not. Note that we discussed finally constraints in the infra meeting... | 21:18 |
| jbernard_ | clarkb: hammer should be fine for now | 21:19 |
| AJaeger | lifeless, http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-03-15-19.03.html | 21:19 |
| *** thorst_ has quit IRC | 21:20 | |
| *** oanson has quit IRC | 21:20 | |
| *** esker has quit IRC | 21:20 | |
| *** esker has joined #openstack-infra | 21:20 | |
| *** ddieterly[away] is now known as ddieterly | 21:20 | |
| lifeless | AJaeger: cool - Nakato ^ | 21:21 |
| *** IlyaG has quit IRC | 21:21 | |
| AJaeger | Nakato, lifeless : Hope we get this done early in Newton... | 21:21 |
| lifeless | AJaeger: bwahahahahahah | 21:22 |
| lifeless | :) | 21:22 |
| *** yamahata has joined #openstack-infra | 21:22 | |
| AJaeger | that's a nice ending for the day ;) | 21:22 |
| * AJaeger waves good-bye | 21:23 | |
| clarkb | fungi: given http://docs.ceph.com/docs/hammer/install/get-packages/#apt how do I take that release.asc and get a fingerprint I can pass to the apt::key puppet module? | 21:23 |
| *** keedya has quit IRC | 21:23 | |
| clarkb | its possible that apt::key will take that file directly too I can check that | 21:23 |
| *** sdake_ is now known as sdake | 21:23 | |
| clarkb | ya looks like that is an option, I suppose we can bikeshed over how silly that is | 21:25 |
| fungi | right, in fact i think we should be using full fingerprints. seems like i already suggested that in another change somewhere? | 21:25 |
| clarkb | oh wait reprepro uses a different module | 21:25 |
| clarkb | oh right becuase root not apt need to trust the key | 21:25 |
| fungi | using too-short key ids is unsafe, and since we don't need to type this over and over we can just put in a fingerprint and call it a day | 21:26 |
| *** esker has quit IRC | 21:26 | |
| *** amitgandhinz has quit IRC | 21:26 | |
| clarkb | that asc is not the fingerprint though its the actual key? | 21:26 |
| fungi | oh, indeed i didn't look at the file--sorry | 21:27 |
| *** baoli has quit IRC | 21:27 | |
| fungi | gpg can render that to a fingerprint for you. so can openssl rsa | 21:27 |
| *** oanson has joined #openstack-infra | 21:27 | |
| *** baoli has joined #openstack-infra | 21:27 | |
| clarkb | gpg --fingerprint $path fails though | 21:27 |
| *** trown|ooobootcam has quit IRC | 21:27 | |
| *** jtomasek has quit IRC | 21:28 | |
| clarkb | looks like I can shove that in public hiera and read it in as a whole though | 21:29 |
| fungi | it may need to be imported into a keyring if you want to use --fingerprint. i'm looking through the manpage for a workflow which doesn't imply an import step | 21:29 |
| Clint | fungi: --no-default-keyring --keyring whateverkey.file --fingerprint or so? | 21:30 |
| clarkb | heh key_id is required | 21:30 |
| clarkb | even though I have the entire key here for consumption | 21:31 |
| fungi | clarkb: if this is the puppet apt module, it at least treats the fingerprint as an id | 21:31 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Archive all of the delorean logs https://review.openstack.org/271416 | 21:31 |
| fungi | really the id is just a truncated fingerprint | 21:31 |
| clarkb | fungi: no its https://github.com/n1tr0g/golja-gnupg because we are updating roots keyring not the apt keyring | 21:31 |
| fungi | oh | 21:31 |
| clarkb | and the fingerprint apepars to work, its just funny that if I provide the entire key contents it still wants a fingerprint/id | 21:32 |
| clarkb | now to figure out what the fingerprint is | 21:32 |
| *** baoli has quit IRC | 21:32 | |
| fungi | openssl rsa -in release.asc -pubout -outform DER | openssl md5 -c | 21:33 |
| fungi | that gives me d4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7e | 21:33 |
| *** thorst_ has joined #openstack-infra | 21:33 | |
| clarkb | d4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7e is what I get | 21:33 |
| clarkb | ok they match | 21:33 |
| clarkb | now to see if thats on a public keyserver I can just fetch from or if I need to vendor the key | 21:34 |
| *** thorst_ has quit IRC | 21:34 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 21:34 |
| *** thorst_ has joined #openstack-infra | 21:34 | |
| *** asettle has joined #openstack-infra | 21:34 | |
| fungi | i'm still working on the equivalent gpg incantation based on Clint's suggestions | 21:35 |
| *** amrith has quit IRC | 21:35 | |
| *** amrith has joined #openstack-infra | 21:35 | |
| Clint | fungi: --with-colons might make that easier | 21:36 |
| *** eharney has joined #openstack-infra | 21:36 | |
| *** pcaruana has quit IRC | 21:38 | |
| clarkb | that key does not apear to be known to mit | 21:38 |
| clarkb | so I think I have to vendor it | 21:38 |
| fungi | Clint: it seems to want to create the keyfile at the path i give it rather than using that | 21:38 |
| *** keedya has joined #openstack-infra | 21:38 | |
| *** jpr has joined #openstack-infra | 21:38 | |
| *** pradk has quit IRC | 21:38 | |
| fungi | clarkb: oh, is this the insane puppet module which insists that you give it a key id so it can retrieve the key from a keyserver network before importing it? we should not use that thing | 21:39 |
| *** sc68cal has quit IRC | 21:39 | |
| fungi | there os zero benefit i can think of to forcing puppet to go download a key when you can provide a key | 21:39 |
| clarkb | fungi: I don't know, it just says key_id is required even when providing the key string directly | 21:39 |
| clarkb | fungi: I am just reusing what mordred has already done and refactoring it to be useable for more than one mirror | 21:40 |
| fungi | yes, i audited this when someone was suggesting using it in our puppet somewhere and said we should look for something that doesn't blindly depend on a keyserver network even though it gets no benefit from using one | 21:40 |
| *** e0ne has quit IRC | 21:40 | |
| fungi | it's as if whoever wrote that module either had an entirely different (and odd) use case in mind, or was blindly copying magic instructions from the internet on how to use openpgp keys | 21:41 |
| *** ZZelle has joined #openstack-infra | 21:41 | |
| lifeless | fungi: ... or both? | 21:41 |
| fungi | lifeless: inclusive or! | 21:41 |
| lifeless | fungi: english! | 21:42 |
| * fungi revises the oed to include xor vs or disambiguation | 21:42 | |
| jeblair | fungi, lifeless: thank you! i wonder why no one thought to do that before! | 21:43 |
| fungi | if people can get "blog" added to the oed, surely we can get "xor" | 21:43 |
| *** baoli has joined #openstack-infra | 21:43 | |
| lifeless | sadly, oed is subscription | 21:44 |
| lifeless | so much for information wants to be free | 21:44 |
| * anteaya goes for a walk | 21:44 | |
| *** maishsk_ has quit IRC | 21:45 | |
| *** baoli has quit IRC | 21:45 | |
| *** baoli has joined #openstack-infra | 21:46 | |
| *** matrohon has quit IRC | 21:47 | |
| *** adrian_otto has quit IRC | 21:49 | |
| *** vgridnev has quit IRC | 21:49 | |
| *** fawadkhaliq has quit IRC | 21:51 | |
| *** fawadkhaliq has joined #openstack-infra | 21:51 | |
| *** oanson has quit IRC | 21:52 | |
| *** pcaruana has joined #openstack-infra | 21:52 | |
| *** mriedem has quit IRC | 21:55 | |
| *** abregman|afk has quit IRC | 21:56 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Make reprepro useable on multiple mirrors https://review.openstack.org/293154 | 21:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Apt mirror for ceph hammer release https://review.openstack.org/293155 | 21:58 |
| clarkb | someone that understands apt and reprepro better than me should review that | 21:58 |
| *** baoli has quit IRC | 21:59 | |
| *** pvaneck has quit IRC | 21:59 | |
| *** ybathia has quit IRC | 22:00 | |
| *** xyang1 has quit IRC | 22:00 | |
| jhesketh | Morning | 22:00 |
| *** yamahata has quit IRC | 22:00 | |
| *** fawadkhaliq has quit IRC | 22:00 | |
| *** woodster_ has joined #openstack-infra | 22:01 | |
| *** fawadkhaliq has joined #openstack-infra | 22:01 | |
| *** harlowja has joined #openstack-infra | 22:01 | |
| *** MarkAtwood has joined #openstack-infra | 22:01 | |
| fungi | lifeless: yeah, i meant wordnet! | 22:01 |
| fungi | though then the suggestion loses a lot of its punch | 22:01 |
| *** baoli has joined #openstack-infra | 22:02 | |
| craige | heya piet. Pholio is certainly there. PLease log in and test it :-) | 22:04 |
| *** tpsilva has quit IRC | 22:07 | |
| lifeless | fungi: :) | 22:07 |
| *** sdake has quit IRC | 22:09 | |
| *** baoli has quit IRC | 22:10 | |
| *** rockyg has quit IRC | 22:10 | |
| *** baoli has joined #openstack-infra | 22:10 | |
| *** dims_ has joined #openstack-infra | 22:11 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:12 | |
| *** flwang1 has quit IRC | 22:12 | |
| *** flwang1 has joined #openstack-infra | 22:12 | |
| *** jpr has quit IRC | 22:13 | |
| *** david-lyle has quit IRC | 22:13 | |
| *** dims has quit IRC | 22:14 | |
| *** gordc has quit IRC | 22:15 | |
| *** piet has quit IRC | 22:16 | |
| *** dims has joined #openstack-infra | 22:16 | |
| *** dims_ has quit IRC | 22:17 | |
| *** claudiub has quit IRC | 22:17 | |
| *** dimtruck is now known as zz_dimtruck | 22:18 | |
| dhellmann | jeblair, fungi, clarkb : can one of you bless this ACL change when you have a few minutes? I'd like to go ahead and start tagging and branching tomorrow, but want the safeguards in place first. https://review.openstack.org/#/c/292781/ | 22:19 |
| *** _david_ has joined #openstack-infra | 22:19 | |
| *** gmurphy has left #openstack-infra | 22:19 | |
| clarkb | dhellmann: would it be possible to let release managers just have those perms always and not use them? | 22:21 |
| clarkb | then we don't have to do this every single release? | 22:22 |
| flwang1 | AJaeger: excuse me, how can i create a new people-group on gerrit for zaqar-ui which is sharing the same core team of zaqar? thanks | 22:22 |
| thiagop | calling it a day, thanks for your help fungi jeblair asselin ianw pabelanger | 22:22 |
| *** keedya has quit IRC | 22:22 | |
| clarkb | flwang1: when you udpate the acl files and put a group in them that does not already exist our tooling will create that group | 22:23 |
| *** thiagop has quit IRC | 22:23 | |
| ianw | pabelanger: so, we got distracted ... we know the builds won't work, but we didn't establish why nodepool didn't even try? | 22:23 |
| clarkb | flwang1: however if the intent is to make the groups the same just reuse the old group | 22:23 |
| fungi | clarkb: in the past it was that we were granting exclusive permissions to release management until the release, and then granting permission to stable management or core teams | 22:23 |
| clarkb | fungi: in this change the perms appear shared | 22:23 |
| flwang1 | clarkb: not same, that's why i asked, since i want to add some horizon guys into the team, thanks | 22:23 |
| clarkb | both $project-release and Release managers get +2 and approve | 22:24 |
| fungi | clarkb: in that case it probably missed the exclusiveGroupPermissions line or forgot the purpose of the change to the steady march of time | 22:24 |
| flwang1 | clarkb: will the tool add me as the 'ptl' so that i can add more people? | 22:24 |
| clarkb | flwang1: not automatically, once the group is created you will need to request that a gerrit admin add the first user who can then add others | 22:24 |
| flwang1 | who is the gerrit admin i should bug? | 22:25 |
| clarkb | flwang1: they answer to infra-root in this channel, you can also just send mail for the request on the infra mailing list | 22:25 |
| flwang1 | clarkb: got, thanks a lot | 22:25 |
| docaedo | howdy - any infra-root folk have a minute to look at something on the app-catalog server for me? | 22:28 |
| jeblair | docaedo: yes | 22:28 |
| docaedo | sweet! There's this script (http://git.openstack.org/cgit/openstack/app-catalog/tree/tools/asset_history.sh) that figures out when any asset was last modified based on git blame | 22:28 |
| *** ddieterly has quit IRC | 22:28 | |
| *** tonytan4ever has quit IRC | 22:28 | |
| docaedo | on production, that script does not seem to find the dates. To confirm, I would like if someone could look for the file: | 22:29 |
| *** piet has joined #openstack-infra | 22:29 | |
| docaedo | openstack_catalog/web/static/assets_merge.yaml | 22:29 |
| docaedo | and confirm that assets_merge.yaml has lots of asset names with no deats | 22:29 |
| docaedo | *dates | 22:30 |
| jeblair | docaedo: where in the filesystem should i find that tree? | 22:30 |
| docaedo | um .. where the repo is checked out, give me a sec and I can find the right path | 22:31 |
| jeblair | well, it's checked out in /opt | 22:31 |
| jeblair | but we usually don't write things to checked out repos | 22:31 |
| jeblair | because that can make git sad in an automated environment | 22:31 |
| *** gnuoy_ has joined #openstack-infra | 22:32 | |
| SamYaple | git reset --hard HEAD # burn it all! | 22:32 |
| jeblair | but apparently we do on the app server! | 22:32 |
| docaedo | normally anything that changes in there from the scripts also cleans up after itself, but I merged a change that would leave that assets_merge.yaml file sitting around | 22:32 |
| *** gnuoy_ has quit IRC | 22:32 | |
| jeblair | docaedo: a todo item might be to leave /opt/apps_site as a pristine git repo, and then install/copy that to some other location | 22:32 |
| *** asettle has quit IRC | 22:33 | |
| docaedo | jeblair: agreed :) | 22:33 |
| jeblair | docaedo: ls /opt/apps_site/openstack_catalog/web/static: assets_dead.yaml assets.schema.yaml assets.yaml blank.json CACHE css images js lib | 22:33 |
| *** MarkAtwood has quit IRC | 22:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add Centos7 wheel cache jobs https://review.openstack.org/290272 | 22:34 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Add a core group for zaqar-ui https://review.openstack.org/293166 | 22:35 |
| *** pvaneck has joined #openstack-infra | 22:35 | |
| *** abregman|afk has joined #openstack-infra | 22:35 | |
| *** asettle has joined #openstack-infra | 22:35 | |
| docaedo | jeblair: ok thanks - that's a little surprising but helps me at least think of some other things to change around | 22:35 |
| jeblair | docaedo: what runs that script? | 22:36 |
| * docaedo find the PDX wifi is not so useful today... | 22:37 | |
| bkero | docaedo: have they fixed their captive portal cert yet? | 22:38 |
| *** _david_ has quit IRC | 22:40 | |
| pabelanger | ianw: no, we don't. We can confirm with clarkb why images didn't seem to queue for building today | 22:42 |
| clarkb | aroo? | 22:42 |
| SamYaple | pabelanger: did my fstrim patch merge? | 22:43 |
| SamYaple | i seem to be breaking infra lately, you should check | 22:43 |
| pabelanger | SamYaple: I am not sure. ianw would know more | 22:43 |
| docaedo | bkero: Maybe? I did not get a warning, but at the same time I seem to be hopping between at least four different APs, so I get connectivity for about a minute, then nothing for another minute... so fun! | 22:43 |
| pabelanger | assuming it is for diskimage-builder | 22:43 |
| clarkb | I don't think dib has released since then | 22:43 |
| SamYaple | pabelanger: sorry i read your statement wrong. i thought you meant you couldnt build images | 22:44 |
| pabelanger | or nodepool-builder hasn't been restarted too | 22:44 |
| *** baoli has quit IRC | 22:44 | |
| pabelanger | SamYaple: last I tested we had no issues | 22:44 |
| bkero | docaedo: yuck :( | 22:44 |
| clarkb | pabelanger: ianw if I had to guess it is because centos-7 is in a busy loop failing to build? | 22:44 |
| *** edmondsw has quit IRC | 22:44 | |
| pabelanger | clarkb: Ya, that is what I think too | 22:45 |
| clarkb | to fix that I think we just need to restart the image build so that it will reread its config | 22:45 |
| pabelanger | if safe, we can try nodepool-builder restart | 22:45 |
| pabelanger | clarkb: good with me doing that then? | 22:46 |
| clarkb | pabelanger: just make sure there aren't any in progress uploads, nodepool image-list | grep building | 22:46 |
| pabelanger | clarkb: sure, 1 sec | 22:46 |
| docaedo | jeblair: just realized no new assets have been merged since the change to keep assets_merge.yaml merged, so that should not have surprised me. But the glamor of air travel calls so I'll try this again tomorrow - thanks for your help! | 22:46 |
| pabelanger | clarkb: Ya, currently centos-7 | 22:47 |
| pabelanger | which is expected | 22:47 |
| jeblair | docaedo: bon voyage | 22:47 |
| clarkb | if that is the only one then ya restart the builder | 22:47 |
| pabelanger | okay, restarted | 22:48 |
| *** weshay has quit IRC | 22:48 | |
| pabelanger | clarkb: ianw: centos-7 building | 22:49 |
| *** hashar has quit IRC | 22:49 | |
| pabelanger | ianw: also, piping to image.log only | 22:49 |
| dhellmann | clarkb : we could, but the point isn't to add permissions, it's to make it so the core team doesn't have them | 22:53 |
| clarkb | dhellmann: right but we don't need to add and remove perms to do that | 22:53 |
| pabelanger | ianw: going to fail because of stale git repos | 22:53 |
| dhellmann | clarkb : how would we do that? | 22:54 |
| clarkb | dhellmann: the existing stable branch acls already exclude the core group | 22:54 |
| pabelanger | fungi: is there anything I can do to fix the project import for gerrit? | 22:54 |
| clarkb | dhellmann: just add your group to that branch prefix and don't bother removing it | 22:54 |
| fungi | clarkb: they want to also exclude the stable branch reviewers though | 22:54 |
| pabelanger | fungi: ianw Oh, maybe not | 22:54 |
| pabelanger | seems to pass | 22:54 |
| clarkb | fungi: but this change does not do that | 22:55 |
| clarkb | at least it doesn't seem to | 22:55 |
| fungi | right, i think the change is probably wrong but i haven't reviewed it yet | 22:55 |
| clarkb | at least for nova | 22:55 |
| dhellmann | clarkb : doesn't it? | 22:55 |
| fungi | still fiddling with election bits | 22:55 |
| dhellmann | clarkb : these also exclude the stable-maint groups, don't they? | 22:56 |
| clarkb | oh I see you are using release/milestone there | 22:56 |
| *** piet has quit IRC | 22:56 | |
| pabelanger | https://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/ is the failure for dib | 22:57 |
| pabelanger | clarkb: fungi: so, is that fixed my manually rerunning manage_project.py on gerrit? | 22:58 |
| clarkb | pabelanger: uh I think we needed to figure out why puppet waasn't running | 22:59 |
| clarkb | then get it to run | 22:59 |
| clarkb | dhellmann: fungi my concern with this is its multile changes per release that have to flow through infra which seems far from ideal. So trying to determine if there is a way we can just set the perms to be what we want and forget about them | 22:59 |
| fungi | yeah, skipping the hard problem to just update the repos isn't a great workaround. we'll just be back to broken as soon as someone merges another new project creation change which is about twice a day on average lately seems like | 22:59 |
| fungi | pabelanger: ^ | 23:00 |
| clarkb | dhellmann: fungi one option would be to use groups that change membership | 23:00 |
| anteaya | clarkb: I think yolanda has said earlier that she was hesitant to run puppet on gerrit since we have two gerrit nodes right now | 23:00 |
| *** rbrndt has quit IRC | 23:00 | |
| clarkb | anteaya: that shouldn't be an issue especially if ansible can only ssh to one of them | 23:00 |
| anteaya | clarkb: and she is uncertain what will happen to the gerrits if puppet runs | 23:00 |
| dhellmann | clarkb, fungi : I agree. I would love to do make this simpler. Can we do that next cycle where we have time to think through the solution? | 23:00 |
| clarkb | dhellmann: sure | 23:00 |
| anteaya | clarkb: I don't know the status of what ansible can ssh into | 23:00 |
| anteaya | clarkb: I just remember her saying that comment while we were in the meeting | 23:01 |
| clarkb | anteaya: neither do I, depends on how yolanda configured the new machine | 23:01 |
| dhellmann | clarkb : groups sounds like a good answer, actually, since that's self-serve for the release team and may even be something we could automate | 23:01 |
| anteaya | clarkb: so that is why puppet is not running, I do believe | 23:01 |
| fungi | clarkb: dhellmann: yes, a gerrit admin manually changing numerous group memberships twice was discussed as the alternative to a project-config core reviewing and approving changes twice | 23:01 |
| fungi | clarkb: dhellmann: but if we put all those groups under the control of the release team instead, then i guess it's up to them to expend that effort | 23:02 |
| dhellmann | if it's scriptable, it's fine with me | 23:02 |
| dhellmann | the question is, how would we do that? create stable-$series groups? or lock all stable branches at the same time? | 23:02 |
| dhellmann | clarkb , fungi : added as a todo in https://etherpad.openstack.org/p/newton-relmgt-plan, maybe we can discuss at the summit | 23:03 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pin puppetlabs-rabbitmq https://review.openstack.org/293074 | 23:04 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test https://review.openstack.org/289445 | 23:04 |
| *** yamamoto_ has quit IRC | 23:05 | |
| ianw | pabelanger: 2016-03-15 22:53:29,483 INFO nodepool.image.build.centos-7: fatal: Couldn't find remote ref master <- it fails on git, as expected | 23:05 |
| pabelanger | fungi: clarkb: okay, so I assume no body stopped it. I can look into the logs and find out why | 23:06 |
| *** dims has quit IRC | 23:06 | |
| clarkb | pabelanger: yolanda dd according to anteaya | 23:06 |
| pabelanger | clarkb: anteaya Ya, I would expect 2 review.o.o to be fine. Especially since ansible would only SSH into 1 | 23:07 |
| clarkb | ya | 23:08 |
| pabelanger | so, guess we hold off until yolanda in the morning. Or look to reenable puppet ourself | 23:08 |
| clarkb | so review.o.o is not in the emergency file | 23:10 |
| clarkb | maybe its in the regular thing but I need to loo kthat up | 23:10 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add debian-jessie image to nodepool https://review.openstack.org/264726 | 23:10 |
| *** bryan_att has quit IRC | 23:12 | |
| clarkb | ansible is definitely not running on review.o.o if I am reading /var/log/puppet_run_all.log correctly | 23:13 |
| clarkb | so its not going and then failing its just not going | 23:14 |
| *** dingyichen has joined #openstack-infra | 23:14 | |
| pabelanger | ya, I haven't looked at puppet logs yet | 23:14 |
| clarkb | review.o.o is in /etc/ansible/hostfile | 23:14 |
| clarkb | and ansible --list-hosts isn't a vlaid thing despite what our docs say | 23:15 |
| fungi | clarkb: the concern is that maybe adding the uuid of the duplicate review.o.o, or perhaps the mere existence of a second review.o.o causing them to both be known only by uuid in the inventory, is causing ansible to skip or break applying to them | 23:16 |
| clarkb | you apparently have to provide a hostslist ? isn't that redudnant? | 23:16 |
| clarkb | fungi: what implies there is a bug in our ansible stuff | 23:16 |
| clarkb | we should be able to ansible one, both, or neither | 23:16 |
| jeblair | there are strange log entries in syslog with out of order dates | 23:17 |
| pabelanger | I would expect that too | 23:17 |
| jeblair | (on review.o.o) | 23:17 |
| clarkb | aha! | 23:18 |
| clarkb | 2016-03-15 23:03:27,321 p=5429 u=root | 87741fa9-92cb-4d1e-8195-6841e2468fb2 : ok=13 changed=6 unreachable=0 failed=0 | 23:18 |
| clarkb | thats running on one of the two review.o.os | 23:18 |
| *** roxanagh_ has joined #openstack-infra | 23:18 | |
| fungi | clarkb: right, that's what i'm saying: maybe there's a bug. we haven't done much with duplicate hosts in the inventory up to this point so it's possible we didn't notice before now that it doesn't work as expected | 23:18 |
| pabelanger | http://puppetdb.openstack.org/node/review.openstack.org seems to confirm that | 23:18 |
| clarkb | now to figure out which of the two that is | 23:19 |
| fungi | also it's possible yolanda's assertion that puppet isn't running on review.o.o is based on puppetboard | 23:19 |
| pabelanger | 104.130.159.134 | 23:19 |
| pabelanger | is that puppetdb is reporting | 23:19 |
| fungi | note that the reports get submitted with whatever identity the inventory has for review.o.o | 23:19 |
| jeblair | it's definitely not running on the prod server | 23:19 |
| clarkb | that uuid belongs to the current prod server | 23:19 |
| jeblair | no entries in syslog since the 10th | 23:19 |
| clarkb | according to nova show | 23:19 |
| pabelanger | o.0 | 23:20 |
| clarkb | oh! | 23:20 |
| EmilienM | crinkle: what are the plans in opnestack-infra to move to puppet4? I'm wondering if Puppet OpenStack CI should do the move in our scenario jobs or not | 23:20 |
| clarkb | crazy theory time | 23:20 |
| * fungi pulls up a chair | 23:20 | |
| clarkb | maybe its puppeting as something other than review.o.o because that uuid doesn't match the site.pp entry? | 23:20 |
| clarkb | eg its just applying the server manifest? though that should break the firewall so maybe not | 23:21 |
| jeblair | clarkb: it should also leave log entries | 23:21 |
| fungi | wouldn't puppet apply still be getting called and log? | 23:21 |
| clarkb | jeblair: good point | 23:21 |
| fungi | yeah, tat | 23:21 |
| crinkle | EmilienM: we haven't made a solid plan yet, but we've inched toward it by adding puppet4 linters | 23:21 |
| jeblair | and login entries that show up in last | 23:21 |
| pabelanger | EmilienM: you could do testing with fedora-23 | 23:21 |
| clarkb | so that uuid isn't in dns at all | 23:21 |
| clarkb | its possible we could be talking to the wrong ip | 23:21 |
| crinkle | pabelanger: well we could also easily do testing with ubuntu-trusty/centos7 using the puppetlabs repos | 23:22 |
| EmilienM | crinkle: I know _ody was about to patch openstack/puppet-openstack-integration to deploy puppet4 instead of 3, but I'm wondering if we need to keep functional testing for 3 or syntax/lint would be enough. | 23:22 |
| clarkb | 104.130.159.134 2001:4800:7818:102:be76:4eff:fe05:9b12 is what it shoud talk to, time to work backwards and see if it is doing that | 23:22 |
| *** roxanagh_ has quit IRC | 23:22 | |
| fungi | we do have that uuid in puppetboard, but unreported http://puppetboard.openstack.org/node/87741fa9-92cb-4d1e-8195-6841e2468fb2 | 23:22 |
| EmilienM | pabelanger: do you mmean puppet4 is in fedora23 ? our jobs are running centos7 and until now, we use puppetlabs packaging. What do you suggest? | 23:23 |
| fungi | ipaddress 104.130.246.91 | 23:23 |
| fungi | ipaddress6 2001:4800:7819:103:be76:4eff:fe05:8525 | 23:23 |
| clarkb | thats what ansible cache says | 23:23 |
| fungi | claims facts were updated 30 minutes ago | 23:23 |
| clarkb | but nova show disagrees? | 23:23 |
| jeblair | fungi: those are the addresses of the new server | 23:23 |
| clarkb | ya I think this is the bug | 23:24 |
| fungi | yep | 23:24 |
| crinkle | EmilienM: i would vote for functional testing with 3 at least for a while longer while it's still fully supported | 23:24 |
| clarkb | ansible inventory is mixing them up and we are puppeting the new server while reporting the uuid of the old server | 23:24 |
| fungi | either that or the way we thought configuring it to skip a host by uuid does the opposite of what we expected | 23:24 |
| EmilienM | crinkle: ok. Then we could create experimental jobs in openstack-infra that would deploy puppet4 at demand | 23:24 |
| clarkb | fungi: I don't think we are skipping but let me double check on that | 23:24 |
| fungi | oh, though it's definitely confusing the uuid of one with the addresses of the other | 23:24 |
| crinkle | EmilienM: ++ | 23:25 |
| fungi | so, yeah, switcheroo. i don't see how incorrect config expectations could account for this at all | 23:25 |
| clarkb | ya that uuid doesn't show up in the disabled stuff anywhere so I think it is just mixing them up | 23:25 |
| *** nwkarsten has quit IRC | 23:25 | |
| pabelanger | EmilienM: right, we default to puppet 4 on fedora, since it is native puppet package | 23:27 |
| EmilienM | pabelanger: awesome. | 23:27 |
| clarkb | I think I found the bug | 23:27 |
| clarkb | patch in a couple | 23:28 |
| fungi | clarkb: shade or in our inventory tooling? | 23:28 |
| *** chlong has joined #openstack-infra | 23:28 | |
| clarkb | our inventory tooling | 23:28 |
| *** fverbs has joined #openstack-infra | 23:28 | |
| clarkb | its a bad list traversal | 23:28 |
| fungi | oh, that's faster to fix at least | 23:28 |
| clarkb | I just have to find out where this file lives in config :) | 23:28 |
| fungi | puppet-ansible? | 23:29 |
| clarkb | /etc/ansible/hosts/openstack | 23:29 |
| fungi | i know i found it not long ago | 23:29 |
| clarkb | is the file I need to edit | 23:29 |
| fungi | oh | 23:30 |
| fungi | no, that's part of ansible i think | 23:30 |
| fungi | mordred: ^ | 23:30 |
| fungi | i believe that's an ansible-openstack component | 23:30 |
| clarkb | found it | 23:31 |
| clarkb | and yes it is in ansible | 23:31 |
| * clarkb puts on the hazard suit and figures out github things | 23:31 | |
| *** adrian_otto has joined #openstack-infra | 23:32 | |
| clarkb | https://github.com/ansible/ansible/blob/devel/contrib/inventory/openstack.py#L152 is the bug fwiw | 23:32 |
| clarkb | I will work on a PR | 23:32 |
| *** IlyaG has joined #openstack-infra | 23:37 | |
| clarkb | my ansible clone is incredibly slow :/ | 23:37 |
| anteaya | :( | 23:37 |
| *** abregman|afk has quit IRC | 23:39 | |
| *** Qiming has joined #openstack-infra | 23:39 | |
| *** ihrachys has quit IRC | 23:45 | |
| *** sdake has joined #openstack-infra | 23:45 | |
| clarkb | https://github.com/ansible/ansible/pull/14984 | 23:47 |
| *** kzaitsev_mb has quit IRC | 23:47 | |
| clarkb | mordred: ^ for when you have a moment | 23:47 |
| *** harlowja has quit IRC | 23:47 | |
| *** eil397 has quit IRC | 23:48 | |
| *** salv-orlando has quit IRC | 23:49 | |
| anteaya | clarkb: nice one | 23:50 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add debian-jessie image to nodepool https://review.openstack.org/264726 | 23:50 |
| *** yamamoto_ has joined #openstack-infra | 23:51 | |
| *** bpokorny has quit IRC | 23:53 | |
| clarkb | I will admit to not know how to test that easily | 23:53 |
| clarkb | we can just give it a go and see what happens :) | 23:53 |
| *** tqtran is now known as tqtran-afk | 23:53 | |
| anteaya | ha ha ha | 23:53 |
| anteaya | that is a way | 23:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test https://review.openstack.org/289445 | 23:58 |
| *** dalgaaf has quit IRC | 23:58 | |
| *** sdake_ has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
