| *** edmondsw has quit IRC | 00:04 | |
| *** mriedem1 is now known as mriedem | 00:04 | |
| *** yamahata has quit IRC | 00:07 | |
| *** ianychoi has joined #openstack-release | 00:14 | |
| *** cody-somerville has quit IRC | 00:19 | |
| *** cody-somerville has joined #openstack-release | 00:20 | |
| *** cody-somerville has quit IRC | 00:20 | |
| *** cody-somerville has joined #openstack-release | 00:20 | |
| *** mriedem has quit IRC | 00:40 | |
| *** sree has joined #openstack-release | 00:41 | |
| *** sree has quit IRC | 00:45 | |
| *** cody-somerville has quit IRC | 01:04 | |
| *** ekcs has quit IRC | 01:05 | |
| *** edmondsw has joined #openstack-release | 01:28 | |
| *** kumarmn has joined #openstack-release | 01:32 | |
| *** kumarmn has quit IRC | 01:43 | |
| *** cody-somerville has joined #openstack-release | 01:43 | |
| *** cody-somerville has quit IRC | 01:43 | |
| *** cody-somerville has joined #openstack-release | 01:43 | |
| *** kumarmn has joined #openstack-release | 01:43 | |
| *** cody-somerville has quit IRC | 01:46 | |
| *** kumarmn has quit IRC | 01:46 | |
| *** kumarmn has joined #openstack-release | 01:47 | |
| *** rosmaita has quit IRC | 01:55 | |
| openstackgerrit | Merged openstack/releases master: Final newton release of Octavia and neutron-lbaas https://review.openstack.org/510621 | 02:01 |
|---|---|---|
| *** hongbin has quit IRC | 02:11 | |
| *** kumarmn has quit IRC | 02:19 | |
| *** thorst has quit IRC | 02:23 | |
| *** thorst has joined #openstack-release | 02:24 | |
| *** thorst has quit IRC | 02:28 | |
| *** dave-mccowan has quit IRC | 02:31 | |
| *** thorst has joined #openstack-release | 02:37 | |
| *** pkovar has quit IRC | 02:50 | |
| *** cody-somerville has joined #openstack-release | 02:58 | |
| *** csomerville has joined #openstack-release | 02:59 | |
| *** cody-somerville has quit IRC | 02:59 | |
| *** bobh has joined #openstack-release | 03:10 | |
| *** kumarmn has joined #openstack-release | 03:20 | |
| *** bobh has quit IRC | 03:20 | |
| *** ricolin has joined #openstack-release | 03:21 | |
| *** lukebrowning_ has joined #openstack-release | 03:23 | |
| *** edmondsw has quit IRC | 03:23 | |
| *** kumarmn has quit IRC | 03:24 | |
| *** lukebrowning has quit IRC | 03:26 | |
| *** hongbin has joined #openstack-release | 03:32 | |
| *** lukebrowning_ has quit IRC | 03:47 | |
| *** lukebrowning has joined #openstack-release | 03:48 | |
| *** cody-somerville has joined #openstack-release | 03:50 | |
| *** csomerville has quit IRC | 03:52 | |
| *** claudiub has joined #openstack-release | 04:07 | |
| *** coolsvap has joined #openstack-release | 04:08 | |
| *** kumarmn has joined #openstack-release | 04:20 | |
| *** edmondsw has joined #openstack-release | 04:21 | |
| *** kumarmn has quit IRC | 04:25 | |
| *** edmondsw has quit IRC | 04:26 | |
| *** lukebrowning_ has joined #openstack-release | 04:27 | |
| *** lukebrowning has quit IRC | 04:29 | |
| *** pcaruana has joined #openstack-release | 04:32 | |
| *** hongbin has quit IRC | 04:40 | |
| *** lukebrowning_ has quit IRC | 04:41 | |
| *** csomerville has joined #openstack-release | 04:42 | |
| *** lukebrowning has joined #openstack-release | 04:42 | |
| *** lukebrowning_ has joined #openstack-release | 04:45 | |
| *** cody-somerville has quit IRC | 04:45 | |
| *** lukebrowning has quit IRC | 04:47 | |
| *** udesale has joined #openstack-release | 04:57 | |
| *** pcaruana has quit IRC | 04:58 | |
| *** armax has quit IRC | 04:59 | |
| *** armax has joined #openstack-release | 05:00 | |
| *** armax has quit IRC | 05:04 | |
| *** jtomasek has joined #openstack-release | 05:04 | |
| *** csomerville has quit IRC | 05:07 | |
| *** yamahata has joined #openstack-release | 05:19 | |
| *** kumarmn has joined #openstack-release | 05:21 | |
| *** csomerville has joined #openstack-release | 05:25 | |
| *** kumarmn has quit IRC | 05:25 | |
| *** csomerville has quit IRC | 05:42 | |
| *** sree has joined #openstack-release | 05:54 | |
| *** udesale__ has joined #openstack-release | 05:55 | |
| *** udesale__ has quit IRC | 05:57 | |
| *** udesale__ has joined #openstack-release | 05:57 | |
| *** udesale has quit IRC | 05:57 | |
| *** jrist has quit IRC | 06:03 | |
| *** sree has quit IRC | 06:07 | |
| *** edmondsw has joined #openstack-release | 06:10 | |
| *** sree_ has joined #openstack-release | 06:11 | |
| *** sree_ is now known as Guest62172 | 06:11 | |
| *** yamamoto has quit IRC | 06:11 | |
| *** edmondsw has quit IRC | 06:14 | |
| *** jrist has joined #openstack-release | 06:16 | |
| *** Guest62172 has quit IRC | 06:20 | |
| openstackgerrit | Thomas Bechtold proposed openstack/releases master: Release renderspec 1.9.1 bugfix release https://review.openstack.org/510764 | 06:37 |
| *** pcaruana has joined #openstack-release | 07:04 | |
| openstackgerrit | Deepthi V V proposed openstack/releases master: release neutron-lib 1.10.1 https://review.openstack.org/510774 | 07:06 |
| *** claudiub|2 has joined #openstack-release | 07:15 | |
| *** claudiub has quit IRC | 07:18 | |
| *** yamahata has quit IRC | 07:18 | |
| *** zen369 has joined #openstack-release | 07:20 | |
| *** kumarmn has joined #openstack-release | 07:23 | |
| *** kumarmn has quit IRC | 07:27 | |
| *** jpich has joined #openstack-release | 07:35 | |
| *** shardy has joined #openstack-release | 07:35 | |
| *** electrofelix has joined #openstack-release | 07:39 | |
| *** amoralej|off is now known as amoralej | 07:49 | |
| *** askb has quit IRC | 07:49 | |
| ttx | The release-post jobs for https://review.openstack.org/510621 failed. tonyb got them reenqueued but they failed with a different error. Should we revert the change or get them retriggered ? | 07:56 |
| *** edmondsw has joined #openstack-release | 07:58 | |
| *** pbandark has joined #openstack-release | 08:00 | |
| *** edmondsw has quit IRC | 08:02 | |
| *** sree has joined #openstack-release | 08:15 | |
| *** sree has quit IRC | 08:20 | |
| *** kumarmn has joined #openstack-release | 08:23 | |
| *** lucas-afk is now known as lucasagomes | 08:25 | |
| tonyb | ttx: Oh phooey I didn't notice they faileda second time. | 08:29 |
| *** kumarmn has quit IRC | 08:29 | |
| ttx | I'll check with fungi if we should reenqueue them from infra side or through revert/resubmit | 08:31 |
| *** inc0 has quit IRC | 08:32 | |
| *** sree has joined #openstack-release | 08:34 | |
| *** askb has joined #openstack-release | 08:37 | |
| *** inc0 has joined #openstack-release | 08:39 | |
| tonyb | ttx: Thanks. | 08:43 |
| smcginnis | We had some out of space issue yesterday. If it's that again, we'll need to wait for his help. | 08:43 |
| tonyb | ttx: Looking at the error and considering the conversation I had with fungi I think revert/reapply shoudl work | 08:45 |
| tonyb | and be safe | 08:46 |
| smcginnis | tonyb: Failure on git fetch on that last one. Yeah, looks like it should be safe to revert/reapply. | 08:51 |
| tonyb | smcginnis: Yeah. | 08:51 |
| *** witek has quit IRC | 09:28 | |
| *** sree has quit IRC | 09:29 | |
| *** thorst has joined #openstack-release | 09:41 | |
| *** edmondsw has joined #openstack-release | 09:46 | |
| *** thorst has quit IRC | 09:47 | |
| *** edmondsw has quit IRC | 09:51 | |
| *** dtantsur|afk is now known as dtantsur | 09:53 | |
| *** witek has joined #openstack-release | 09:53 | |
| *** alexchadin has joined #openstack-release | 09:54 | |
| *** ricolin has quit IRC | 09:54 | |
| *** wolverineav has joined #openstack-release | 09:55 | |
| *** tbarron|PTO is now known as tbarron | 10:04 | |
| *** wolverineav has quit IRC | 10:07 | |
| *** lukebrowning_ has quit IRC | 10:11 | |
| *** sdague has joined #openstack-release | 10:12 | |
| *** lukebrowning has joined #openstack-release | 10:22 | |
| *** kumarmn has joined #openstack-release | 10:26 | |
| *** lukebrowning has quit IRC | 10:27 | |
| *** wolverineav has joined #openstack-release | 10:30 | |
| *** kumarmn has quit IRC | 10:30 | |
| *** alexchadin has quit IRC | 10:46 | |
| *** alexchadin has joined #openstack-release | 10:47 | |
| *** dtantsur has quit IRC | 11:04 | |
| *** alexchadin has quit IRC | 11:07 | |
| *** dtantsur has joined #openstack-release | 11:08 | |
| *** dtantsur has quit IRC | 11:08 | |
| *** dtantsur has joined #openstack-release | 11:09 | |
| *** dtantsur has quit IRC | 11:09 | |
| *** dtantsur has joined #openstack-release | 11:13 | |
| *** lucasagomes is now known as lucas-hungry | 11:15 | |
| *** alexchadin has joined #openstack-release | 11:16 | |
| *** lukebrowning has joined #openstack-release | 11:23 | |
| *** rosmaita has joined #openstack-release | 11:24 | |
| *** kumarmn has joined #openstack-release | 11:27 | |
| *** sree has joined #openstack-release | 11:30 | |
| *** kumarmn has quit IRC | 11:31 | |
| *** sree has quit IRC | 11:34 | |
| *** ricolin has joined #openstack-release | 11:36 | |
| *** yamamoto has joined #openstack-release | 11:38 | |
| *** pkovar has joined #openstack-release | 11:41 | |
| *** yamamoto has quit IRC | 11:44 | |
| *** yamamoto has joined #openstack-release | 11:44 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** yamamoto has joined #openstack-release | 11:48 | |
| *** yamamoto has quit IRC | 11:49 | |
| *** yamamoto has joined #openstack-release | 11:55 | |
| *** ricolin has quit IRC | 12:00 | |
| *** edmondsw has joined #openstack-release | 12:03 | |
| *** edmondsw has quit IRC | 12:04 | |
| *** edmondsw_ has joined #openstack-release | 12:05 | |
| *** pkovar has quit IRC | 12:06 | |
| *** sree has joined #openstack-release | 12:07 | |
| *** thorst has joined #openstack-release | 12:16 | |
| *** alexchadin has quit IRC | 12:17 | |
| ttx | smcginnis: ok let me propose that | 12:19 |
| dhellmann | o/ | 12:20 |
| openstackgerrit | Thierry Carrez proposed openstack/releases master: Revert "Final newton release of Octavia and neutron-lbaas" https://review.openstack.org/510860 | 12:22 |
| *** pkovar has joined #openstack-release | 12:23 | |
| openstackgerrit | Thierry Carrez proposed openstack/releases master: Revert "Revert "Final newton release of Octavia and neutron-lbaas"" https://review.openstack.org/510861 | 12:25 |
| ttx | smcginnis: here you go ^ | 12:25 |
| *** lucas-hungry is now known as lucasagomes | 12:27 | |
| *** yamamoto has quit IRC | 12:27 | |
| *** kumarmn has joined #openstack-release | 12:27 | |
| *** yamamoto has joined #openstack-release | 12:29 | |
| *** kumarmn has quit IRC | 12:32 | |
| *** ricolin has joined #openstack-release | 12:32 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** alexchadin has joined #openstack-release | 12:39 | |
| *** udesale__ has quit IRC | 12:44 | |
| *** thorst_ has joined #openstack-release | 12:49 | |
| *** thorst has quit IRC | 12:49 | |
| fungi | wait, so that releases ref failed a second time after i reenqueued it? | 12:51 |
| fungi | when did we have an out of space issue yesterday (and where)? | 12:52 |
| fungi | but yeah, i agree http://logs.openstack.org/85/858c988956fce8ac8ed8a202fe91aadbc0387395/release-post/tag-releases/25848a5/console.html#_2017-10-10_02_22_54_676966 does look like a network connectivity problem (similar to why i reenqueued it to begin with) | 12:55 |
| fungi | and both the node and the merger are in the same region of rackspace, so either our git server farm was under extreme load or they're having internal network connectivity problems | 12:56 |
| *** zhurong has joined #openstack-release | 12:58 | |
| *** shardy has quit IRC | 13:00 | |
| *** edmondsw_ is now known as edmondsw | 13:01 | |
| *** kzaitsev1pi has quit IRC | 13:05 | |
| *** pkovar has quit IRC | 13:05 | |
| *** pkovar has joined #openstack-release | 13:06 | |
| *** kzaitsev_pi has joined #openstack-release | 13:06 | |
| *** kumarmn has joined #openstack-release | 13:06 | |
| *** shardy has joined #openstack-release | 13:11 | |
| *** oanson has quit IRC | 13:14 | |
| *** oanson has joined #openstack-release | 13:15 | |
| *** mriedem has joined #openstack-release | 13:22 | |
| *** lbragstad has joined #openstack-release | 13:28 | |
| *** shardy has quit IRC | 13:29 | |
| *** bobh has joined #openstack-release | 13:33 | |
| *** dave-mccowan has joined #openstack-release | 13:33 | |
| *** kumarmn has quit IRC | 13:35 | |
| *** kumarmn has joined #openstack-release | 13:35 | |
| *** kumarmn has quit IRC | 13:35 | |
| *** kumarmn has joined #openstack-release | 13:36 | |
| *** kumarmn has quit IRC | 13:38 | |
| *** shardy has joined #openstack-release | 13:38 | |
| *** kumarmn has joined #openstack-release | 13:38 | |
| smcginnis | fungi: Wasn't the tarball signing failure yesterday an out of space issue? Maybe i misunderstood. | 13:39 |
| fungi | smcginnis: no, not out of space | 13:39 |
| fungi | to save on quota in our control plane tenant, we replaced half our zuul v2 mergers (zm05-08) with v3 mergers | 13:40 |
| fungi | however, the persistent v2 job nodes had git remotes cached from prior to that rejiggering | 13:40 |
| fungi | so you can see in the log zuul-cloner was told to use zm03 (a v2 merger) but then raised an error about zm07 (now a v3 merger) | 13:40 |
| fungi | more evidence that the (deprecated and replaced with a no-op shim in v3) zuul-cloner utility was designed to assume single-use node lifecycles | 13:40 |
| fungi | [resent from my input history for this channel] | 13:41 |
| smcginnis | Ah, so not out of space. Wrong space. :) | 13:42 |
| fungi | indeed ;) | 13:43 |
| fungi | the fix was to just delete all the old workspaces so they'd get cloned anew when the job reran | 13:43 |
| openstackgerrit | Merged openstack/releases master: Revert "Final newton release of Octavia and neutron-lbaas" https://review.openstack.org/510860 | 13:44 |
| *** zhurong has quit IRC | 13:45 | |
| smcginnis | fungi: Is this another one? http://logs.openstack.org/3d/3d5c1125a92cd90e739816ce01d72b9811c60521/release-post/tag-releases/6fe4bee/console.html#_2017-10-10_13_45_28_065698 | 13:48 |
| openstackgerrit | Merged openstack/releases master: Revert "Revert "Final newton release of Octavia and neutron-lbaas"" https://review.openstack.org/510861 | 13:50 |
| fungi | smcginnis: seems like it's failing consistently... i wonder if there's something wrong with the way the job is setting up the gerrit remote to be able to access the git notes | 13:51 |
| fungi | i'll hop into that workspace and take a look | 13:51 |
| fungi | it's likely that some unexercised behavior change crept into that job because we had the remotes persisting with the workspace from a previous run | 13:51 |
| smcginnis | Yeah, revert of the revert just hit the same thing: http://logs.openstack.org/b8/b8d320d6c94f8b1cc51a1336be3e574d10883b17/release-post/tag-releases/34a40b2/console.html | 13:52 |
| fungi | and so we didn't expose the problem until after i wiped the cached workspaces and it had to reset it from scratch | 13:52 |
| fungi | yeah, it has no gerrit remote at all and is, i think, trying to retrieve notes refs from the origin (in this case the zuul merger) | 13:54 |
| *** hongbin has joined #openstack-release | 13:54 | |
| fungi | did this job previously run git review -s or something maybe? | 13:55 |
| fungi | and perhaps that ended up getting dropped at some point? | 13:55 |
| fungi | if so, that would explain the behavior | 13:55 |
| *** ricolin has quit IRC | 13:56 | |
| smcginnis | No remote? Guess that would explain why git fetch isn't happy. | 13:58 |
| *** armax has joined #openstack-release | 13:58 | |
| fungi | yeah, it gets normal refs from the zuul merger set as its origin, but that won't include git notes, which only gerrit serves up | 13:59 |
| *** d0ugal has quit IRC | 14:00 | |
| *** d0ugal has joined #openstack-release | 14:00 | |
| *** d0ugal has quit IRC | 14:00 | |
| *** d0ugal has joined #openstack-release | 14:00 | |
| *** d0ugal has quit IRC | 14:01 | |
| *** d0ugal has joined #openstack-release | 14:02 | |
| *** d0ugal has joined #openstack-release | 14:02 | |
| ttx | boo | 14:02 |
| fungi | or maybe i'm mistaken and we do include the notes refs in replicas, but if so the logic here is wrong http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/release-tools/release_from_yaml.sh#n71 | 14:02 |
| ttx | can't wait until I revert the revert of the revert of the revert | 14:02 |
| *** alexchadin has quit IRC | 14:03 | |
| fungi | yeah, it works if i change the origin from zm03.o.o to git.o.o | 14:06 |
| *** oanson has quit IRC | 14:06 | |
| fungi | so we do replicate git notes to our push mirrors, but zuul mergers don't pull notes refs | 14:06 |
| *** oanson has joined #openstack-release | 14:07 | |
| fungi | one working hack is to add a second remote to the git farm and then set that for the notes refs | 14:11 |
| dhellmann | fungi : I don't fully understand all of the differences in the way zuul is now setting up git repos, but it seems like the changes have broken a lot of assumptions in the release tools so I should probably look into it. Is there any documentation I can look at? | 14:11 |
| fungi | one working hack is to add a second remote to the git farm and then set that for the notes refs | 14:12 |
| fungi | er, up-arrow at the wrong moment | 14:12 |
| fungi | dhellmann: what, you mean in zuul v2? | 14:12 |
| fungi | this is a v2 job problem | 14:12 |
| dhellmann | I don't really know what's changed, but this all *used* to work so something in the past couple of weeks is different. | 14:12 |
| fungi | just happened to be latent (likely for many months) exposed when i flushed the old copies of the workspaces on signing01.ci | 14:13 |
| dhellmann | so what was the change that was exposed when you did that flushing? | 14:13 |
| dhellmann | was that a zuul-cloner change? or a change to our mirror setup? something else? | 14:13 |
| dhellmann | the git portion of our infra is sort of a grey box to me :-/ | 14:13 |
| fungi | the job is configured to clone the releases repo if it doesn't exist. at some point it managed to add a remote to its git config which provided git notes refs | 14:14 |
| fungi | which the job itself has been relying on | 14:14 |
| dhellmann | that would have been a very very early thing -- we've always used the notes to get the info about who approved a release | 14:15 |
| dhellmann | I guess I also assumed that these jobs always cloned new copies of the repos every time | 14:15 |
| fungi | however, since the repository was getting reused rather than completely deleted and recreated on every run, _something_ subsequently changed for the git config setup but since that was never exercised (due to reusing the copy on the persistent node it always runs on) we never noticed | 14:15 |
| dhellmann | yeah, I'm trying to understand that "_something_" :-) | 14:16 |
| dhellmann | it sounds like it may not be easy to tell exactly what it was | 14:16 |
| dhellmann | given the time that has passed | 14:16 |
| *** alexchadin has joined #openstack-release | 14:16 | |
| fungi | my guess is that the job originally did its own cloning, perhaps from git.openstack.org or review.openstack.org (which _do_ provide notes refs), then at a later date switched to using zuul-cloner which retrieves the ref from our zuul mergers (which don't pull notes refs), but since the origin was already set in .git/config under that job's workspace it kept working | 14:17 |
| *** thorst_ has quit IRC | 14:18 | |
| dhellmann | ah | 14:19 |
| fungi | and this is simply the first time we've wiped workspaces on the signing01.ci.o.o node | 14:19 |
| fungi | since the job switched to using zuul-cloner | 14:19 |
| dhellmann | that may be it; I remember a period where we had a bunch of raw git commands doing clones | 14:19 |
| fungi | so anyway, i've temporarily "fixed" the origin remote in that job's workspace such that it should go back to tits prior behavior for now | 14:20 |
| dhellmann | so, what's the solution? add notes to what we mirror? have the job add a separate remote that include the notes? | 14:20 |
| dhellmann | I thought the workspaces were cleared but zuul-cloner was pulling from a local mirror that was kept up to date. Is that incorrect? | 14:21 |
| dhellmann | *cleared automatically every time | 14:21 |
| fungi | the proper solution, if zuul v2 weren't basically frozen, would be to make sure zuul mergers also pull git notes refs and serve them, and improve zuul-cloner to be a bit more thorough on reused systems (it's designed more with single-use nodes and green-field workspaces in mind) | 14:22 |
| fungi | for v3, i mainly just need to double-check that zuul is actually pushing notes refs along with the repo state, or we need to look at altering the job to add a second remote explicitly for notes refs | 14:23 |
| dhellmann | and for zuul v3 I guess we get the workspace cleanup because we won't need to use a static node, but we'll have even more trouble with the notes because there's not even an origin remote (IIUC) so we're going to have to update this job for that case anyway | 14:23 |
| fungi | er, well first remote in that case, but yes that's more or less the situation | 14:23 |
| fungi | so anyway, the job as it stands on v2 should work for now by way of me hacking the workspace back into some semblance of its prior latent configuration | 14:24 |
| fungi | and i'll be double-checking the v3 story around git notes | 14:25 |
| dhellmann | ok. I would like to be involved in any fix for v3, since that sounds like a good opportunity to learn more about how these bits work under the new system. | 14:25 |
| fungi | but feel free to approve that re-re-revert at your convenience | 14:25 |
| *** edmondsw has quit IRC | 14:25 | |
| dhellmann | also because, you know, I'll *help* with the fix :-) | 14:25 |
| fungi | dhellmann: you bet, and help is appreciated! | 14:25 |
| *** edmondsw has joined #openstack-release | 14:26 | |
| *** edmondsw_ has joined #openstack-release | 14:27 | |
| *** edmonds__ has joined #openstack-release | 14:28 | |
| *** edmondsw has quit IRC | 14:30 | |
| *** zen369 has quit IRC | 14:31 | |
| *** thorst has joined #openstack-release | 14:31 | |
| *** edmondsw_ has quit IRC | 14:31 | |
| *** claudiub|2 has quit IRC | 14:33 | |
| *** alexchadin has quit IRC | 14:37 | |
| smcginnis | ttx: Did you want the fun of reverting the revert of the reverting revert> | 14:37 |
| ttx | I'm in a meeting tunnel now | 14:38 |
| ttx | so... no | 14:38 |
| smcginnis | OK, I can take care of that. | 14:38 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Revert "Revert "Revert "Final newton release of Octavia and neutron-lbaas""" https://review.openstack.org/510904 | 14:39 |
| *** nikhil has joined #openstack-release | 14:44 | |
| *** ricolin has joined #openstack-release | 14:57 | |
| *** thorst has quit IRC | 15:01 | |
| *** kumarmn has quit IRC | 15:22 | |
| *** thorst has joined #openstack-release | 15:23 | |
| *** kumarmn has joined #openstack-release | 15:23 | |
| openstackgerrit | Merged openstack/releases master: Revert "Revert "Revert "Final newton release of Octavia and neutron-lbaas""" https://review.openstack.org/510904 | 15:26 |
| *** kumarmn has quit IRC | 15:27 | |
| *** edmonds__ has quit IRC | 15:29 | |
| *** thorst has quit IRC | 15:30 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Revert "Revert "Revert "Revert "Final newton release of Octavia and neutron-lbaas"""" https://review.openstack.org/510916 | 15:31 |
| *** udesale__ has joined #openstack-release | 15:31 | |
| johnsom | lol | 15:31 |
| *** shardy_ has joined #openstack-release | 15:32 | |
| *** edmondsw has joined #openstack-release | 15:32 | |
| *** udesale__ has quit IRC | 15:33 | |
| smcginnis | johnsom: Let's just hope we don't end up with more than 5 "revert"s in the commit message. ;) | 15:33 |
| *** thorst has joined #openstack-release | 15:34 | |
| johnsom | Yep. Thanks for tracking this down for us! I still wonder if we should not have selected a canary as our mascot. | 15:34 |
| smcginnis | Haha, that would have been a good one. | 15:34 |
| smcginnis | Looks like the post-release jobs all passed on the last revert, so the redo revert should hopefully work this time. | 15:36 |
| *** edmondsw_ has joined #openstack-release | 15:36 | |
| *** edmondsw has quit IRC | 15:37 | |
| *** thorst has quit IRC | 15:39 | |
| *** shardy_ has quit IRC | 15:40 | |
| *** edmondsw_ has quit IRC | 15:41 | |
| openstackgerrit | Merged openstack/releases master: Revert "Revert "Revert "Revert "Final newton release of Octavia and neutron-lbaas"""" https://review.openstack.org/510916 | 15:42 |
| *** thorst has joined #openstack-release | 15:43 | |
| *** yamahata has joined #openstack-release | 15:44 | |
| smcginnis | Woot, tag-releases passed. | 15:44 |
| *** thorst_ has joined #openstack-release | 15:46 | |
| *** thorst has quit IRC | 15:49 | |
| johnsom | +1 | 15:50 |
| *** edmondsw has joined #openstack-release | 15:50 | |
| smcginnis | Post jobs complete. Looks like we are in the clear. For now. | 15:51 |
| *** kumarmn has joined #openstack-release | 15:51 | |
| *** kumarmn_ has joined #openstack-release | 15:53 | |
| *** thorst has joined #openstack-release | 15:53 | |
| *** csomerville has joined #openstack-release | 15:53 | |
| *** thorst_ has quit IRC | 15:54 | |
| *** edmondsw has quit IRC | 15:55 | |
| *** kumarmn has quit IRC | 15:56 | |
| *** edmondsw has joined #openstack-release | 15:56 | |
| *** yamahata has quit IRC | 16:00 | |
| *** dave-mcc_ has joined #openstack-release | 16:01 | |
| *** dave-mccowan has quit IRC | 16:04 | |
| *** pcaruana has quit IRC | 16:07 | |
| *** lucasagomes is now known as lucas-afk | 16:08 | |
| *** spzala has joined #openstack-release | 16:13 | |
| *** spzala has quit IRC | 16:13 | |
| openstackgerrit | Emilien Macchi proposed openstack/releases master: TripleO Ocata release https://review.openstack.org/510931 | 16:21 |
| *** kumarmn has joined #openstack-release | 16:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/releases master: TripleO Pike release https://review.openstack.org/510933 | 16:25 |
| *** kumarmn_ has quit IRC | 16:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/releases master: TripleO Ocata release https://review.openstack.org/510931 | 16:32 |
| *** kumarmn has quit IRC | 16:35 | |
| *** kumarmn has joined #openstack-release | 16:36 | |
| *** jpich has quit IRC | 16:39 | |
| *** csomerville has quit IRC | 16:41 | |
| *** kumarmn has quit IRC | 16:41 | |
| *** csomerville has joined #openstack-release | 16:41 | |
| *** kumarmn has joined #openstack-release | 16:41 | |
| *** shardy has quit IRC | 16:46 | |
| EmilienM | release-managers: please review https://review.openstack.org/#/c/510933/ and https://review.openstack.org/#/c/510931/ | 16:53 |
| *** kumarmn has quit IRC | 16:56 | |
| toabctl | smcginnis, hey. could you merge https://review.openstack.org/#/c/510764/ please? | 16:56 |
| *** kumarmn has joined #openstack-release | 16:56 | |
| smcginnis | toabctl: On its way. | 16:56 |
| toabctl | smcginnis, thanks! | 16:57 |
| smcginnis | EmilienM: I'll take a look at yours next. | 16:57 |
| EmilienM | smcginnis: thanks | 16:57 |
| dhellmann | if things are working now, I can take a look at the queue | 16:57 |
| dhellmann | it is my review day... | 16:57 |
| smcginnis | dhellmann: Yep, we should be in the clear now. | 16:57 |
| smcginnis | dhellmann: If you wouldn't mind, I was just about to grab some lunch. | 16:57 |
| dhellmann | sure, I'm on it | 16:58 |
| openstackgerrit | Merged openstack/releases master: Release renderspec 1.9.1 bugfix release https://review.openstack.org/510764 | 17:02 |
| *** inc0 has quit IRC | 17:02 | |
| *** inc0 has joined #openstack-release | 17:03 | |
| *** lifeless has quit IRC | 17:03 | |
| *** kumarmn has quit IRC | 17:04 | |
| *** tellesnobrega has joined #openstack-release | 17:05 | |
| openstackgerrit | Merged openstack/releases master: TripleO Ocata release https://review.openstack.org/510931 | 17:10 |
| openstackgerrit | Merged openstack/releases master: TripleO Pike release https://review.openstack.org/510933 | 17:12 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: add release notes output to the list-changes job https://review.openstack.org/509278 | 17:14 |
| *** lifeless has joined #openstack-release | 17:20 | |
| *** kumarmn has joined #openstack-release | 17:21 | |
| *** lukebrowning has quit IRC | 17:24 | |
| *** lukebrowning has joined #openstack-release | 17:24 | |
| *** kumarmn has quit IRC | 17:25 | |
| *** ricolin has quit IRC | 17:25 | |
| *** zen369 has joined #openstack-release | 17:34 | |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| *** cody-somerville has joined #openstack-release | 17:52 | |
| *** kumarmn has joined #openstack-release | 17:52 | |
| *** csomerville has quit IRC | 17:54 | |
| *** sree has quit IRC | 17:59 | |
| *** sree has joined #openstack-release | 18:00 | |
| *** sree has quit IRC | 18:05 | |
| *** jtomasek has quit IRC | 18:09 | |
| *** amoralej is now known as amoralej|off | 18:13 | |
| *** pkovar has quit IRC | 18:15 | |
| *** coolsvap has quit IRC | 18:15 | |
| *** electrofelix has quit IRC | 18:21 | |
| *** kumarmn has quit IRC | 18:28 | |
| *** kumarmn has joined #openstack-release | 18:28 | |
| *** kumarmn has quit IRC | 18:33 | |
| *** cody-somerville has quit IRC | 18:53 | |
| *** cody-somerville has joined #openstack-release | 18:55 | |
| openstackgerrit | Lance Bragstad proposed openstack/releases master: Add keystone deadlines to queens schedule https://review.openstack.org/509835 | 19:00 |
| fungi | dhellmann: https://review.openstack.org/510994 Use a separate git remote for release job notes | 19:11 |
| fungi | that's my suggestion for the workaround | 19:11 |
| fungi | we can iterate from there | 19:11 |
| fungi | i confirmed that zuul does not (in v2 nor v3) provide git notes refs (yet) | 19:11 |
| fungi | for the future, i found how i believe you're supposed to be able to access them from gitpython | 19:12 |
| fungi | so we can probably get them pushed into the prepared repos on the job nodes with a little bit of additional work in zuul | 19:13 |
| *** armax has quit IRC | 19:14 | |
| dhellmann | fungi : thanks, that looks good with 1 comment | 19:27 |
| fungi | dhellmann: good call, thanks! | 19:29 |
| fungi | will fix unless you want to do the honors | 19:29 |
| dhellmann | go for it | 19:29 |
| *** pkovar has joined #openstack-release | 19:41 | |
| *** thorst has quit IRC | 19:46 | |
| *** thorst has joined #openstack-release | 19:51 | |
| fungi | dhellmann: looks like the variable you're suggesting gets set to the full path on the filesystem: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/release-jobs.yaml#n19 | 19:53 |
| dhellmann | aw, bummer | 19:53 |
| fungi | should i attempt to parse the repository name out of that path? | 19:53 |
| fungi | (and can we assume local users will always set it to the same name?) | 19:53 |
| dhellmann | no, we probably can't make that assumption | 19:54 |
| fungi | alternatively, it may make sense to just hard-code openstack/releases there in the script, with the expectation that we'll have to fix that (along with a bunch of other release stuff probably) when we flatten namespaces | 19:54 |
| dhellmann | well, I know we can't, because I'd break it myself | 19:54 |
| dhellmann | does the local repo have any other remotes at all? | 19:54 |
| dhellmann | would those match except for the hostname? | 19:55 |
| dhellmann | in the reno jobs we were finding that the repo didn't have a remote under v3 | 19:55 |
| fungi | in v2 it has an origin remote which is the zuul merger, but in v3 it has no remotes initially until we add one | 19:55 |
| dhellmann | right | 19:55 |
| dhellmann | ok, so let's just hard-code it for now | 19:55 |
| fungi | can do | 19:55 |
| *** pkovar has quit IRC | 19:55 | |
| *** thorst has quit IRC | 19:56 | |
| dhellmann | oh, I guess you could use ZUUL_PROJECT if it's set | 19:56 |
| dhellmann | fungi : ^^ | 19:56 |
| dhellmann | that way at least the CI system would keep working after the rename | 19:56 |
| dhellmann | it should be pretty clear what the error is when someone runs it locally | 19:56 |
| clarkb | there is a compatibility role to have the ZUUL_* things added to your env but they aren't part of the v3 default api like they were in v2 | 19:56 |
| dhellmann | maybe even throw in a warning "ZUUL_PROJECT is not set, defaulting releases repo to xxx" | 19:57 |
| fungi | dhellmann: that could work | 19:57 |
| dhellmann | clarkb : the 2 cases we have for this script are running in CI and running on my local server (and possibly ttx's or smcginnis') | 19:57 |
| fungi | clarkb: yeah, i anticipate needing to tweak playbooks/release/tag.yaml to add $ZUUL_PROJECT | 19:58 |
| dhellmann | oh, I see what you mean now | 19:58 |
| fungi | since this was one of the release jobs mordred attempted to rewrite as a v3 native job and i'd rather not revert it to legacy just for one compatibility variable | 19:58 |
| dhellmann | yeah, we can also change ZUUL_PROJECT to some other name if there's a different variable | 19:59 |
| smcginnis | So default it to work right locally but fix it to work right in the gate. | 19:59 |
| mordred | fungi, smcginnis, dhellmann: tl;dr me? | 19:59 |
| fungi | smcginnis: yeah, that's what i expect we're ending up with | 19:59 |
| dhellmann | smcginnis : right. although "work right locally" will stop working right when we flatten the git namespaces, but at that point attempting to update the repo will fail with an error that should make it clear the URL is bad | 19:59 |
| fungi | mordred: https://review.openstack.org/510994 | 20:00 |
| fungi | commit message is the tl;dr | 20:00 |
| smcginnis | dhellmann: I have a feeling there will be some pain when that happens anyway. | 20:00 |
| dhellmann | likely | 20:01 |
| mordred | fungi: cool - that | 20:05 |
| mordred | fungi: what if we added the remote in the pre-playbook of the tag-releases job rather than updating that jenkins script? | 20:05 |
| mordred | fungi: that way we could actually use the canonical_name from the zuul variables | 20:06 |
| dhellmann | mordred : that wouldn't solve the local user case, would it? | 20:06 |
| dhellmann | I guess we could do both | 20:06 |
| mordred | sure it would - the local user already has an origin - so if we just leave it as origin ... | 20:07 |
| smcginnis | Don't we already have the remote locally? | 20:07 |
| dhellmann | mordred : oh, sure, that makes sense | 20:07 |
| mordred | and have tag-releaess just add an origin remote to every project that's in {{ zuul.projects }} using canonical name | 20:07 |
| dhellmann | that works | 20:08 |
| *** thorst has joined #openstack-release | 20:08 | |
| dhellmann | is that a thing we just turn on, or is it a separate script? | 20:08 |
| mordred | something like this: http://paste.openstack.org/show/623275/ | 20:10 |
| dhellmann | swanky | 20:10 |
| fungi | mordred: the script itself is still going to break as-is because it explicitly calls `git remote update origin` | 20:14 |
| mordred | fungi, dhellmann, smcginnis: https://review.openstack.org/511004 Add an origin remote for tag-releases for notes <-- there is a sake-of-argument version | 20:14 |
| fungi | mordred: oh, i see. you're adding a remote called origin | 20:14 |
| mordred | yah. mostly wanting to avoid mucking with that script during the interregnum | 20:15 |
| fungi | mordred: the other incentive for my change was to correct the job under v2 in case we end up having to roll back again, since it's only working right now by way of manual surgery i performed to the workspace on signing01.ci | 20:15 |
| mordred | fungi: ah - gotcha - that makes sense to me | 20:16 |
| fungi | but i'm willing to acknowledge that's a low risk (if we roll back to v2 we just need to avoid blowing away the workspaces there again until we're finally on v3) | 20:16 |
| dhellmann | fungi : maybe we should do both things, and update the script to first check for an origin remote then check for the notes | 20:16 |
| fungi | i think i'm in favor of mordred's change, since it's less fiddling with scripts which are basically working for now, and the risk of this breaking again even if we roll back to v2 is reasonably low now that we know why it broke | 20:18 |
| mordred | also - as soon as we're happy that we're not going to rollback, this particular job is a GREAT example to work on to figure out a good story for wanting both local execution and remote execution | 20:19 |
| mordred | since part of the theory behind using ansible for job content is that theoretically one should be able use playbook content that you use for other things also in zuul jobs | 20:19 |
| dhellmann | fungi : ok, that makes sense | 20:19 |
| mordred | but most of our content so far is content writen specifically for zuul jobs | 20:19 |
| dhellmann | fungi , mordred , smcginnis : were we going to be able to move the definition of the job into the releases repo? or does it need to stay in a more shared repo? | 20:20 |
| mordred | it seems there's a good opportunity to learn what 'run it locally' means here | 20:20 |
| mordred | dhellmann: we can totally move it to the releases repo - and I think we should | 20:20 |
| fungi | dhellmann: one up-side to v3 is that it does allow us to keep the secrets these jobs need under infra control while letting the jobs which use them move wherever | 20:20 |
| dhellmann | I usually check out the releases repo to the commit with the stuff I want processed then run the script and point it at that directory. It figures out which deliverable files changed by using git commands. | 20:20 |
| mordred | dhellmann: there are no jobs that must be in project-config - the only thing thatmust go into project-config is assigning a job to another project | 20:21 |
| fungi | dhellmann: the not-so-well-explained idea with zuul v3 is that you should be able to check out your repo and then run your jobs (by just calling ansible and telling it the job you want to run) | 20:21 |
| mordred | fungi: yah - that's a thing we need some better stories on | 20:22 |
| smcginnis | Should we prioritize moving in-repo? Or we can take time to move things over once things have settled, right? | 20:22 |
| fungi | smcginnis: the latter, in my opinion | 20:22 |
| smcginnis | fungi: OK, that's what I thought. | 20:22 |
| mordred | dhellmann: so tag-releases can be in openstack/releases (or openstack/release-tools) - along with the secrets it needs - along with the project-template for it | 20:23 |
| mordred | but assigning that project-template to all of the official repos has to happen in project-config | 20:23 |
| mordred | it'll really be a question of whether the ssh key (needed to push tags to gerrit) and the launchpad api key are secrets that want to be owned by the release team or the infra team | 20:24 |
| fungi | yeah, i suppose we could always have an infra-root encrypt the secrets those jobs need and then submit changes to the releases repo or wherever containing the encrypted secret material | 20:24 |
| mordred | (the job using a secret must be in the repo where the secret is defined - so if infra owns the secret infra has to own the job) | 20:24 |
| mordred | fungi: nope - we can't | 20:24 |
| fungi | we can't? | 20:24 |
| dhellmann | mordred : ideally we'll be able to stop using release-tool soon | 20:24 |
| mordred | fungi: oh - I see what you're saying - yes, we could do that | 20:25 |
| dhellmann | we're already moving some of the other commands into the releases repo | 20:25 |
| mordred | fungi: but the release team would be able to see them by submitting a change to print the secret, so I tihnk we'd want to shift ownership of the secret | 20:25 |
| fungi | mordred: agreed, that is a risk we have to take into account | 20:25 |
| dhellmann | does that mean anyone would be able to submit a patch and print the contents of the key? | 20:27 |
| dhellmann | or would we have special privileges because we have +2 on the repo? | 20:28 |
| *** openstackgerrit has quit IRC | 20:33 | |
| *** sree has joined #openstack-release | 20:36 | |
| *** sree has quit IRC | 20:40 | |
| *** wolverineav has quit IRC | 20:59 | |
| *** zen369 has quit IRC | 21:01 | |
| fungi | dhellmann: secrets are scoped to post-review pipelines only | 21:13 |
| fungi | so a core reviewer would at least need to approve a change to the repo containing the secret to expose it | 21:14 |
| dhellmann | so someone would have to approve a patch | 21:14 |
| dhellmann | got it | 21:14 |
| *** claudiub|2 has joined #openstack-release | 21:15 | |
| *** thorst has quit IRC | 21:21 | |
| *** yamahata has joined #openstack-release | 21:26 | |
| *** edmondsw has quit IRC | 21:33 | |
| *** edmondsw_ has joined #openstack-release | 21:35 | |
| *** edmondsw_ has quit IRC | 21:39 | |
| *** edmondsw has joined #openstack-release | 22:08 | |
| *** bobh has quit IRC | 22:11 | |
| *** edmondsw has quit IRC | 22:13 | |
| *** pbandark has quit IRC | 22:17 | |
| *** yamamoto has joined #openstack-release | 22:19 | |
| *** sdague has quit IRC | 22:39 | |
| *** armax has joined #openstack-release | 22:48 | |
| *** sree has joined #openstack-release | 23:01 | |
| *** sree has quit IRC | 23:05 | |
| *** claudiub|2 has quit IRC | 23:15 | |
| *** thorst has joined #openstack-release | 23:22 | |
| *** hongbin has quit IRC | 23:22 | |
| *** thorst has quit IRC | 23:27 | |
| *** rosmaita has quit IRC | 23:34 | |
| *** mriedem has quit IRC | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!