Wednesday, 2019-04-10

*** jamesmcarthur has joined #zuul01:20
*** jamesmcarthur has quit IRC01:23
*** openstackgerrit has quit IRC01:30
*** openstackgerrit has joined #zuul01:45
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Dockerfile: add support for arbritary uid  https://review.openstack.org/65142401:45
*** bhavikdbavishi has quit IRC02:11
tristanCmordred: with ^ i was able to add nodepool services to the zuul-operator. it's almost feature complete for integration in software factory now :)02:48
tristanCmordred: would be nice to get the project created to have ci and image publication in place02:53
*** bhavikdbavishi has joined #zuul02:57
clarkbtristanC: the &> is a bashism iirc. May want to use 2>&1 so that the shebang is valid on images where sh is not bash03:14
clarkbalso if you set user at build time which that change does why do you need to dynamically update /etc/passwd?03:16
clarkbcouldnt you write /etc/passwd at build time and avoid the entrypoint entirely?03:16
clarkbor is the arbitrary uid important and maybe we drop the user dorective at build time?03:18
tristanCclarkb: k8s may not respect image user config when using securityContext03:22
tristanCclarkb: build time user is for local runtime usage, not sure why user isn't added to passwd at build time though03:23
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Dockerfile: add support for arbritary uid  https://review.openstack.org/65142403:24
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Dockerfile: add support for arbritary uid  https://review.openstack.org/65024603:24
clarkbI see so the uid may be arbitray in some.cases but we also want to set it to not root for the other cases03:27
tristanCclarkb: it seems like a reasonable thing to not run zuul/nodepool as root for the other cases too, but that's optional03:28
clarkbya I have no opposition to that. Just making sure I understand why we need the USER directive and the entrypoint03:28
clarkbat first glance they seem mutually exclusive03:29
clarkbbut it is because the USER set uid may be overridde03:29
clarkbI cant vote now because I am on my phonr but I'll leave a review in the morning03:29
tristanCclarkb: thanks!03:30
*** openstackstatus has quit IRC04:35
*** openstackstatus has joined #zuul04:37
*** ChanServ sets mode: +v openstackstatus04:37
*** quiquell|off is now known as quiquell|rover04:48
*** pcaruana has joined #zuul05:06
*** bjackman_ has joined #zuul05:26
*** bhavikdbavishi has quit IRC05:54
*** bhavikdbavishi has joined #zuul05:55
*** quiquell|rover is now known as quique|rover|brb06:28
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to node  https://review.openstack.org/63733806:38
*** gtema has joined #zuul06:47
*** sshnaidm|pto is now known as sshnaidm07:06
*** quique|rover|brb is now known as quiquell|rover07:11
*** pcaruana has quit IRC07:34
*** pcaruana has joined #zuul07:35
openstackgerritMerged openstack-infra/zuul master: tox: Fix indentation  https://review.openstack.org/64904007:48
*** jpena|off is now known as jpena07:52
*** quiquell|rover is now known as quique|rover|bbl07:54
openstackgerritLuigi Toscano proposed openstack-infra/zuul-jobs master: stage-output: fix dotfiles handling  https://review.openstack.org/64865807:58
openstackgerritLuigi Toscano proposed openstack-infra/zuul-jobs master: stage-output: fix dotfiles handling  https://review.openstack.org/64865808:05
openstackgerritMerged openstack-infra/zuul master: Assert ara database is loaded from ansible venv  https://review.openstack.org/65091708:09
*** bhavikdbavishi has quit IRC08:24
*** themroc has joined #zuul08:29
*** hashar has joined #zuul08:38
*** hashar has quit IRC08:39
*** bhavikdbavishi has joined #zuul09:04
*** electrofelix has joined #zuul09:28
*** quique|rover|bbl is now known as quiquell|rover09:50
*** bhavikdbavishi has quit IRC10:53
*** jpena is now known as jpena|lunch11:34
*** bhavikdbavishi has joined #zuul11:52
*** rlandy has joined #zuul11:55
*** rlandy is now known as rlandy|ruck11:56
*** hashar has joined #zuul12:01
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Dockerfile: add support for arbritary uid  https://review.openstack.org/65024612:19
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Dockerfile: add support for arbritary uid  https://review.openstack.org/65142412:20
*** jpena|lunch is now known as jpena12:31
openstackgerritTobias Henkel proposed openstack-infra/zuul master: WIP: Support emitting warnings via zuul_return  https://review.openstack.org/65152612:54
*** bjackman_ has quit IRC12:57
*** yolanda_ has joined #zuul12:58
pabelangerHmm, I think might have a bug with noop jobs and github returning the wrong ref_url: https://dashboard.zuul.ansible.com/t/ansible/build/b144ba131f364e5887f5509aa1dd5c5112:59
pabelangernote is says https://api.github.com/repos/ansible-network/resource_module_builder/pulls/24 but should maybe be https://github.com/ansible-network/resource_module_builder/pulls/2412:59
pabelangersorry: https://github.com//ansible-network/resource_module_builder/pull/2413:00
pabelangertobiash: you seen that by chance?^13:00
*** irclogbot_0 has joined #zuul13:04
*** altlogbot_2 has joined #zuul13:08
tobiashpabelanger: I never looked at the build results of noop ;)13:13
*** pwhalen has joined #zuul13:30
*** bhavikdbavishi has quit IRC13:33
*** quiquell|rover is now known as quiquell|lunch13:38
*** yolanda_ has quit IRC13:49
*** quiquell|lunch is now known as quiquell|rover14:02
Shrewsclarkb: the squashed change lgtm. just a few unnecessary whitespace changes in a couple of files14:12
Shrewshopefully we can try that again today, though i'm going to be in and out with a couple of appointments today14:13
*** openstackgerrit has quit IRC14:14
clarkbI'll be around shortly ina spot to watch it if we give it another go14:14
AJaegerclarkb: please self-approve once you're ready - I'll +2 now...14:15
*** hashar has quit IRC14:22
Shrewshttps://review.openstack.org/651339 helps stabilize nodepool tests if anyone has a moment today. easy review14:38
clarkbthat eill make a good test of the image builds14:43
clarkbI'll review it once image change is in14:43
*** ianychoi has quit IRC14:56
*** quiquell|rover is now known as quiquell|off15:01
clarkbmordred: tobiash fungi if you can review https://review.openstack.org/#/c/651339/1 and if it looks good +2 but don't +3 I can add the approval vote once the image roles update15:01
tobiashclarkb: lgtm15:03
mordredclarkb: ++15:06
*** openstackgerrit has joined #zuul15:08
openstackgerritMerged openstack-infra/zuul-jobs master: Only alias registries on build nodes  https://review.openstack.org/65139315:08
mordredclarkb: ^^ woot15:10
clarkbcool approving the other change now15:11
*** gtema has quit IRC15:11
*** jangutter_ is now known as jangutter15:15
*** themroc has quit IRC15:15
clarkbI think skopeo has the same issue15:16
clarkblocalhost | time="2019-04-10T15:15:48Z" level=fatal msg="Invalid source name docker://[2607:ff68:100:54:f816:3eff:fef2:fc69]:5000/zuul/nodepool:latest: invalid reference format"15:16
clarkbmordred: ^15:16
clarkbI bet everyone is importing that docker distribution/reference lib regex and using it15:17
clarkbbecause it is the "standard"15:17
clarkbit just completely fails at ipv6ing15:17
openstackgerritClark Boylan proposed openstack-infra/zuul-jobs master: Revert "Only alias registries on build nodes"  https://review.openstack.org/65157915:22
clarkbShrews: AJaeger mordred ^ fyi15:22
AJaegerclarkb: I'm reaching out to "my" skopeo expert now...15:23
clarkbAJaeger: https://github.com/moby/moby/issues/39033 may be relevant15:28
*** jamesmcarthur has joined #zuul15:32
AJaegerclarkb: which skope command do you use? Can you leave an example and I link to this discussion...15:33
clarkbAJaeger: http://logs.openstack.org/39/651339/1/gate/nodepool-upload-image/143d1ce/ara-report/file/2bdc78fb-bf65-4b40-a93a-3decdda15759/#line-1 that is the command that fails15:34
mordredclarkb: AWESOME15:34
clarkbfwiw the docker push commands also failed15:35
fungiconsistency!15:35
* fungi sighs15:35
clarkbbut I think we half expected skopeo to work so that is the more noteworthy item :)15:35
clarkbI think we may want to go back to the drawing board on this one15:46
openstackgerritMerged openstack-infra/zuul-jobs master: Revert "Only alias registries on build nodes"  https://review.openstack.org/65157915:46
clarkb/etc/hosts isn't a good hack for anything on localhost and skopeo not working means we need something for localhost15:46
clarkband we want to avoid localhost edits of /etc/hosts because we don't want executors in untrusted contexts to be able to ssh back into the executor somehow and I like /etc/hosts being read only as a layer of protection there15:47
clarkbReal dns is likely massive overkill for these ephemeral setups15:48
clarkbwe could create an ipv4 label :/15:49
clarkbor put effort into fixing https://github.com/moby/moby/issues/3903315:50
clarkbAJaeger: my hunch is that because the urls are docker:// skopeo parses them using the docker rules which don't allow the [] escapes. But I bet docker:// is how you tell skopeo to talk to a docker registry15:54
clarkbmaybe ^ hints at another option here. Run a different registry with different push/pull tool15:54
mordredthe docker demons on nodes need to be able to talk to the registries15:55
mordredotherwise the jobs aren't good at testing that docker images work as they would in the wild15:55
clarkbmordred: hrm thats a fair point15:55
clarkband we also want ot push to dockerhub15:55
mordredyeah15:56
mordredso - I think effort in to fixing the moby issue potentially - or yeah, an ipv4 label which would be sadmaking15:56
clarkbwe could potentially use the intermediate registry for everything15:58
clarkbthat may have scaling problems but could have permanent real dns record that made ipv6 happier15:58
AJaegerclarkb: timing is bad, I'll relay once I hear something from my colleague15:59
clarkbno worries I don't think this is so much a rush as me just throwing ideas out in a fit of brainstorming15:59
clarkbI guess I can make a PR to fix the regex issue in docker16:01
clarkbI'll start on that now16:01
clarkbat the very least maybe that will get the wheels spinning upstream16:01
mordredclarkb: the problem with using the intermediate registry for everything is that on the consume side jobs want to just be able to say "I want zuul/zuul:latest" and have that result in the correct speculative state for that buildset16:01
mordredclarkb: that said - we COULD spin up a dns server on the buildset registry host and configure the remote hosts to talk to it via dns. of course, that still doesn't help with localhost16:02
clarkbmordred: ya, but we do know who are parent change(s) are and could be smarter about pulling just their images I think16:03
* mordred is mostly thinking out loud16:03
clarkbs/are/our/16:03
*** bhavikdbavishi has joined #zuul16:03
mordredclarkb: it's important that job content be able to pull images themselves16:03
mordredclarkb: such as the docker-compose one16:03
mordredclarkb: I think the upstream moby patch will be super helpful16:04
clarkbmordred: ya I'm forking the docker distribution repo now16:04
clarkbthe regex is actually quite simple. I just need to figure out how to write it in go16:05
clarkbshould have it up shortly16:05
*** jamesmcarthur has quit IRC16:06
*** jamesmcarthur has joined #zuul16:08
pabelangerso, I've noticed a few times that zuul misses github events, I actually haven't looked yet to see where they could be dropping. EG: github side, network or zuul side.  However, I am wondering if there might be a way to audit open PRs for a project, and if there was no zuul comment after X mins, to leave a recheck comment, or some other process to try and enqueue said PR.  The idea would be to not have users16:28
pabelangerkeep track if zuul didn't run a PR, then do the recheck themself16:28
pabelangerOh, so zuul did get the event from github16:31
pabelangerbut raised an exception16:31
pabelangerhttp://paste.openstack.org/show/749130/16:31
tobiashpabelanger: permission problem?16:32
pabelangertobiash: nope, zuul has been gating that repo for a while16:33
tobiashwas that pr created?16:33
pabelangeryes16:33
mordredmaybe race condition on the GH side16:33
pabelangerI can get the payload from github app16:33
mordredlike - it sent the payload but when we went to query things about it it wasn't showing up yet16:34
pabelangerOh, maybe16:34
mordredpabelanger: also, per your earlier thing, I think obviosly fixing exceptions is important :) - then we've got some stuff sketched out related to HA scheduler that should allow us to start running more than one event receiver to allow not missing stuff16:34
pabelangermordred: Yah, nice, cool. I'll fix exception first, since that is the issue here16:35
mordredwhich is to say - I think first priority should be to make not missing thigns in the first place rock solid before we start having to poll things regularly to see what might have been missed ... but I agree, it should not be on the users to try to keep track of this in their heads16:35
mordredlike - that's a terrible experience for the users for sure :)16:35
mordredpabelanger: maybe it just needs a retry/backoff something something?16:35
pabelangeragree, and yah I think a retry should be okay16:36
mordred++16:36
pabelangerlet me see if I can write up a patch16:36
mordred\o/16:36
tobiashbut we need to be careful there as this loop is time critical16:36
mordredyeah16:36
tobiashso retrying is only a good idea for a few seconds16:36
mordredtobiash: yup. although we should only ever need to retry if gh-side is backed up somehow - getting a NotFound at that point in the logic should be very uncommon, one would hope16:37
mordredand I'm guessing more common on public github than on a private ghe16:37
tobiashyes, except you really have a permission issue16:37
tobiashwhich can be easily the case if not using app auth16:38
mordredah - this is a good point16:38
pabelangerwe actually aready retry, upto 5 times it looks like in that function, we just don't see to catch the exception16:40
pabelangerso, maybe we just need to catch it and hope we get it on next loop16:40
pabelangerhttp://git.zuul-ci.org/cgit/zuul/tree/zuul/driver/github/githubconnection.py#n114116:41
pabelangertobiash: what do you think?16:41
tobiash++16:42
tobiashI think the target of that loop probably was exactly catching this case16:42
tobiash(and possibley infrastructure related issues)16:42
mordredyeah16:43
tobiashso when you catch that exception, we should not only catch NotFoundError16:43
*** altlogbot_2 has quit IRC16:45
tobiashmordred: as you did the initial implementation of the command module override in zuul this might be interesting for you: https://review.openstack.org/65027616:48
mordredtobiash: yeah- I keep looking at that16:48
tobiashthanks16:49
mordredtobiash: do you know what impact, if any, this has on ordering between stdout and stderr streams?16:49
mordredtobiash: one of the reason we were originally combining them is to get the closest approximation of "foo.sh 2>&1" - so that log lines would be interleaved appropriately. I'm guessing that the select([]) call is gonna do mostly the same?16:50
tobiashmordred: I think that should be mostly the same16:51
mordrednod16:53
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Handle project not found exception better  https://review.openstack.org/65160916:54
pabelangertobiash: mordred: how about ^16:54
pabelangeroh, some backscoll, reading now16:54
pabelangertobiash: ah, so catch all exceptions...16:55
tobiashYepp ;)16:55
mordredpabelanger: otherwise I think that looks solid16:56
pabelangerk, fixing16:57
pabelangeris, except github3.exceptions good enough there?16:57
tobiashI think yes16:57
pabelangergithub3.exceptions.GitHubException seems to be base16:59
tobiashk16:59
pabelangerall others parent to it16:59
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Handle github exceptions better when trying to get a PR  https://review.openstack.org/65160917:06
clarkbhttps://github.com/docker/distribution/pull/2899 ok that took entirely too much time17:06
clarkbmordred: AJaeger Shrews ^ fyi17:07
pabelangermordred: tobiash: ^updated17:07
*** jpena is now known as jpena|off17:07
pabelangerclarkb: might have thoughts too^17:07
clarkbpabelanger: reviewed. Just a fix for the log string (and commit message update)17:28
pabelangerfixing17:33
openstackgerritMerged openstack-infra/zuul master: Fix zuul-stream functional test  https://review.openstack.org/65029717:34
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Handle github exceptions better when trying to get a PR  https://review.openstack.org/65160917:36
pabelangerupdated17:36
clarkb+2 thanks17:37
pabelangernp!17:37
pabelangerty17:37
pabelangermordred: tobiash: if you'd like to review again^17:38
mordred++17:38
mordredpabelanger: thanks!17:38
pabelangeralso, for spacex fans, another falcon heavy launch today! about 4 hours out17:39
clarkb6:30 Eastern is whne launch windows starts roughly iirc17:40
clarkbpabelanger: this one has customer payload too17:40
pabelangeryah, that was for webcast starting17:40
pabelangerclarkb: yah, should be awesome17:41
*** jamesmcarthur has quit IRC17:42
*** jamesmcarthur has joined #zuul18:12
*** brennen has left #zuul18:13
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Support emitting warnings via zuul_return  https://review.openstack.org/65152618:21
*** electrofelix has quit IRC18:24
*** tobiash has quit IRC18:30
*** tobiash has joined #zuul18:30
*** tobiash has quit IRC18:35
*** tobiash has joined #zuul18:40
openstackgerritMerged openstack-infra/nodepool master: Fix race with test_provider_removal unit test  https://review.openstack.org/65133918:42
*** tobiash has quit IRC18:46
*** hashar has joined #zuul18:49
*** tobiash has joined #zuul18:51
*** themroc has joined #zuul18:56
-openstackstatus- NOTICE: Restarting Gerrit on review.openstack.org to pick up new configuration for the replication plugin19:06
*** rlandy|ruck is now known as rlandy|ruck|brb19:41
pabelangerthe other thing I have noticed, is sometime as soon as I create a PR, I apply by gate label, because it is a trival fix.  Most of the time, once check jobs report back, it will get enqueued into gate. But every so often, zuul won't move it to gate.  Again, I haven't looked in the logs... mostly just sharing my thoughts :)19:43
*** bhavikdbavishi has quit IRC19:44
fungishout out to zuul for its ability to test the insane velocity of openstack development through yet another major coordinated release! http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004906.html19:48
pabelanger++19:49
fungimerging an *average* of 155 changes a day19:50
fungithat's more than a change every 10 minutes for ~6 months19:51
fungi(not considering how many patchsets on average go into each change, how many jobs are run on average for each change, et cetera)19:51
pabelangeryah, when I looked this morning openstack peaked at 1.2 kjps19:52
fungithere are only a handful of software efforts in the world, floss or proprietary, approaching that scale19:52
fungiearlier today the openstack release team basically pushed a button and then watched as zuul.openstack.org built, signed, published and announced release artifacts for dozens of complex services and many more supporting modules. the only failures were in generating release notes for two who hadn't upgraded off a deprecated sphinx theme19:57
fungistart to finish was roughly 30 minutes19:57
fungithis is an activity which only a few years ago when openstack was a fraction its current size took days of hand coordination19:58
*** jamesmcarthur has quit IRC19:59
pabelangerYah, the cool part for me, is starting to make those bits more generic for any zuul operator to consume (release team).  I looked into it a few months back, just haven't dug more into it20:00
fungii'd love to see more projects start providing their own release signing keys20:02
fungifor openstack it's already just a zuul secret20:02
fungimaster key is kept on independent hardware, rotated twice annually, dedicated signing subkey is generated and encrypted as a zuul secret kept in the tenant's config project20:03
fungiopenstack uses the same signing subkey to sign its generated git tags and source tarballs20:05
fungiall from a handful of lines of configuration20:05
*** rlandy|ruck|brb is now known as rlandy|ruck20:09
fdegirmordred: i'm looking for your zuul presentation to present Zuul to my team20:21
fdegirmordred: they aren't available on your personal website - do you have it somewhere where I can get?20:21
fdegirand this is perhaps a more general comment - would it be possible to put zuul architecture diagram to zuul website so people can see things on a high level?20:23
fdegirsomething like this: https://storage.googleapis.com/cdn.thenewstack.io/media/2018/05/bc0977f2-zuul-screen-02-2.jpg20:23
clarkbfdegir: a similar diagram should be on the website20:23
clarkblet me find a link20:23
* fdegir needs to improve his search skills...20:24
fdegirclarkb: do you mean the one on the page https://zuul-ci.org/docs/zuul/admin/components.html ?20:25
clarkbhttps://zuul-ci.org/docs/zuul/admin/components.html20:26
clarkbya that20:26
fdegiri like the ascii one more but this works too ;)20:26
fdegirthanks20:27
mordredfdegir: https://opendev.org/inaugust/inaugust.com/src/branch/master/src/zuulv3/test-the-future.rst21:04
mordredfdegir: is the latest intro version - but there are several versions of the talk in that directory21:04
fdegirmordred: thanks!21:07
mordredfdegir: (I need to do a better job organizing them for purpose)21:08
pabelangermordred: clarkb: https://review.openstack.org/649371/ actually updates it to add optional db21:09
*** hashar has quit IRC21:32
*** themroc has quit IRC21:32
SpamapScomponents is missing the SQL db21:56
SpamapSpabelanger:derp, your answer was better21:57
openstackgerritMerged openstack-infra/zuul master: Add database info for components graph  https://review.openstack.org/64937122:30
clarkbpabelanger: SpamapS ^ fyi22:39
pabelangerwoot22:43
mnaseris there a generic ansible-lint job available?22:44
clarkbI want to say now beacuse we have a bunch of different rule exclusions in ours and pin to < 4?22:46
clarkbs/now/no/22:46
mnaserclarkb: gotcha23:01
SpamapSyo dawg, I heard you like ansible so I wrote an ansible role to lint your ansible roles.23:13
pabelangermnaser: I usually just wrap it in tox.ini as linters entry point23:14
mnaseryeah I figured as much23:15
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job  https://review.openstack.org/53208323:21
mnaseroh good timing23:22
mnaser:P23:22
tristanCmnaser: you might be interested by this one too https://review.openstack.org/#/c/530682/, we are using it a lot to provide generic linters without bothering with tox23:25
mnasertristanC: cool, I'll check it out23:26
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver  https://review.openstack.org/64947423:36
SpamapS^^ going to try this out before it merges anyway, but, that is kind of killing us at the moment and I"m not even sure it works.. the test is a little bit thin23:37
SpamapSI actually think it doesn't work, because it doesn't read the total number of servers already running. :-P23:37
* SpamapS is sure the test is deeply flawed23:37
*** ianychoi has joined #zuul23:46
*** rlandy|ruck has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!