Wednesday, 2018-10-17

*** rlandy is now known as rlandy|bbl00:13
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add overview of config options  https://review.openstack.org/60498400:40
*** ssbarnea has joined #zuul02:25
*** ssbarnea|bkp2 has quit IRC02:25
*** tobiash has quit IRC02:32
*** tobiash has joined #zuul02:34
*** bhavikdbavishi has joined #zuul03:41
pabelangerwoot03:42
pabelangergot mazer build to work03:42
pabelangerwith a few depends-on and help of zuul :)03:42
pabelangerhttps://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/24/24/f6a46af6b655657d239fd40b258b12ae92b69fa5/check/build-ansible-role-tarball/90fe219/job-output.html#l46503:43
pabelangernow I sleep03:43
*** pcaruana has quit IRC04:46
openstackgerritMerged openstack-infra/zuul master: Add role path to JSON result in callback plugin  https://review.openstack.org/60710504:53
openstackgerritMerged openstack-infra/zuul master: Simplify Builds page  https://review.openstack.org/61013704:56
openstackgerritMerged openstack-infra/zuul master: Use zuul/web/static in setup_hook  https://review.openstack.org/60976405:52
*** rlandy|bbl is now known as rlandy06:02
*** mmedvede has quit IRC06:15
*** mmedvede has joined #zuul06:16
tobiashcorvus: the zuul-quick-start job is unstable. It looks like a timing issue to me: http://logs.openstack.org/65/602665/2/gate/zuul-quick-start/8c87892/job-output.txt.gz#_2018-10-17_05_15_46_62857006:32
*** pcaruana has joined #zuul06:34
*** hashar has joined #zuul07:05
openstackgerritMerged openstack-infra/zuul master: Improve docs for inventory_file  https://review.openstack.org/60266507:09
*** themroc has joined #zuul07:13
*** jpena|off is now known as jpena07:30
*** themroc has quit IRC07:31
*** electrofelix has joined #zuul07:59
*** nilashishc has quit IRC08:21
AJaegerZuul team, https://zuul.openstack.org/builds is broken, it's empty ;(08:45
AJaegertobiash, corvus, mordred, did the changes merged this morning break it?08:46
tobiashAJaeger: that might be true08:47
tobiashAJaeger: but I have no clue why, that shouldn't be the case08:47
tobiashAJaeger: it's a browser cache issue08:48
tobiashwhen I clean the cache it works08:48
tobiashAJaeger: it could be that the browser cached the index.html which referenced the hash-named js files that are now replaced by different hash-named js files during deployment08:49
tobiashmordred, tristanC, corvus: ^08:50
tobiashshift-reload didn't fix it for me (probably because of the service worker). I had to really delete the cache.08:51
AJaegertobiash: indeed, works now - thanks!08:53
*** themroc has joined #zuul08:53
tobiashmordred, tristanC, corvus: to me it looks like the service worker currently has more problems than advantages so we at least should think about wether we should disable it completely and rely on plain old browser caching instead09:04
evrardjpAJaeger: tobiash I deleted my cache, still broken when doing a query like looking for osa in project09:04
evrardjphttp://zuul.openstack.org/builds?project=openstack%2Fopenstack-ansible09:04
tobiashevrardjp: that indeed throws an exception that is related to a code change09:08
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Revert "Simplify Builds page"  https://review.openstack.org/61126609:11
tobiashAJaeger, infra-root: if this needs to be fixed now ^09:11
AJaegertobiash: I'm fine with waiting...09:16
tobiashk09:16
*** goern has joined #zuul09:18
evrardjptobiash: wait I cleaned my browser cache for nothing? Disaster! I have to re-log-in everywhere!09:27
evrardjp:p09:27
tobiashevrardjp: cache != cockies ;)09:28
tobiashShit mobile keyboard...09:29
evrardjphahaha09:32
evrardjpI removed cache, then removed cookies, then removed everything, then ... but it doesn't matter it was just for lightening the mood... explaining jokes is not fun!09:33
evrardjptobiash: I am curious about your mobile keyboard now. What did we avoid? :p09:34
*** bhavikdbavishi has quit IRC09:46
fricklerI'm seeing an annoying "feature" on the zuul status page: if I middle-click on a review link in order to open it in a new tab (which IMO should happen by default and would make my life much easier), firefox does open the new tab as intended, but at the same time the state of the box is toggled, expanding/unexpanding the list of running jobs for that patch09:53
fricklersecond question: is there a way to filter builds by result != SUCCESS? I'd like to see also e.g. POST_FAILURE on https://zuul.openstack.org/builds?pipeline=gate&result=FAILURE09:55
openstackgerritMerged openstack-infra/zuul master: Indicate whether a build is voting in the logs  https://review.openstack.org/60585710:22
lennybHi, I have ssh issue with zuul 2.5.2 multi gerrit connection. I've configured 2 gerrit connections and I am getting events from both, but I see that it uses same SSH connection for both gerrits in #link https://github.com/openstack-infra/zuul/blob/2.5.2/zuul/connection/gerrit.py#L38210:37
*** sshnaidm has joined #zuul11:05
*** panda is now known as panda|lunch11:18
*** pcaruana has quit IRC11:28
*** jpena is now known as jpena|lunch11:37
*** sshnaidm has quit IRC11:48
*** panda|lunch is now known as panda12:03
tobiashcorvus, mordred: what do you think about splitting the queued state in the status ui in something like waiting-for-parent and waiting-for-job (in shorter better words)?12:04
*** ssbarnea has quit IRC12:05
tobiashbackground is that many of our projects use job dependencies and from watching the status ui I cannot distinguish if a job waits for a node (which should be short) or for a parent job to succeed12:05
*** ssbarnea has joined #zuul12:07
tobiashso currently I'm looking often at many queued jobs and don't know if that's just normal (because of dependencies) of if there is a capacity problem12:07
tobiashlennyb: I'm not sure if multi gerrit was a real and tested thing in zuul212:15
lennybtobiash: yeah, I was afraid of this.... I saw an example in zuul.conf12:19
tobiashlennyb: maybe it was but I don't know12:19
tobiashlennyb: do you have plans for zuulv3?12:19
* tobiash doesn't miss zuulv212:20
lennybtobiash: not for the near future. we have a lot of jobs written in yaml for JJB and currently it looks like a big effort to move this to zuul312:21
tobiashlennyb: looking back into my zuulv2 days we in fact used two gerrit connections so we had that working12:23
tobiashlennyb: but I have to check if I had patches on master regarding this back these days12:24
*** pcaruana has joined #zuul12:24
lennybtobiash, thanks.12:25
tobiashlennyb: it's a shot into the dark but maybe this might help you: https://review.openstack.org/40612912:27
lennybtobiash: thanks, looks like exactly the issue that I am facing12:30
tobiash:)12:30
*** rlandy has joined #zuul12:32
lennybtobiash: I am getting into https://review.openstack.org/#/c/406129/4/zuul/scheduler.py line 1027, since my connection_name remains of the first gerrit connection and not the second one12:44
*** jpena|lunch is now known as jpena12:44
*** chandankumar is now known as chkumar|off13:03
openstackgerritMerged openstack-infra/nodepool master: Use zuul-sphinx for configuration layout  https://review.openstack.org/60427413:22
openstackgerritMerged openstack-infra/nodepool master: Add overview of config options  https://review.openstack.org/60498413:23
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Don't fetch ready nodes when satisfying min-ready requests  https://review.openstack.org/61133014:14
tobiashShrews: follow up to optimize ready node fetching ^14:14
Shrewsk14:15
Shrewstobiash: could you just squash that into the first? i don't want to merge the first without it14:19
tobiashShrews: sure, no problem14:20
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines  https://review.openstack.org/61049514:20
tobiashShrews: ^14:21
Shrewsthx. i'll approve it after seeing the tests14:21
tobiashk14:21
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment  https://review.openstack.org/61049614:22
*** bhavikdbavishi has joined #zuul14:28
*** bhavikdbavishi has quit IRC14:29
*** bhavikdbavishi has joined #zuul14:29
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Extract pep8 messages for inline comments  https://review.openstack.org/58963414:30
mordredcorvus: ^^ added a failed_when: false ... turns out the look-for-failures code didn't run if the tox task failed :)14:31
*** rlandy_ has joined #zuul14:32
mordredtobiash: re: job dependencies and status - that doesn't seem like a bad idea to me. to make things more complex - I wonder how impossible it would be to actually visually show the dependency graph within the change box14:33
mordredtobiash: I have no ideas either of how that would look or how to code it14:34
corvusyeah, i think finer-grained queued status is a good idea.  so is a visual job graph in the box, but, having written the last visual graph... oh boy.  :)14:34
tobiashmordred: the easy solution would not be a dependency graph but just a 'this job is waiting for *a* parent' and 'this job is queued and waiting for a node'14:35
corvusfrickler: i think a fix to your click issue merged recently... maybe some amount of cache clearing is necessary for you to get the new js code?14:35
dmsimardmordred: tristanC wrote a job and pipeline graph14:35
corvusdmsimard: we're talking about on the status page14:36
*** openstackgerrit has quit IRC14:36
dmsimardsure, what I meant is that in terms of code it shouldn't be that much different14:37
mordredtobiash: ++14:37
dmsimardhttps://review.openstack.org/#/c/537871/ && https://review.openstack.org/#/c/537871/14:37
dmsimarder14:37
dmsimardhttps://review.openstack.org/#/c/537869/14:37
mordreddmsimard: yah - I tink the visual design of how to fit something like that into the status box is the hard bit here14:38
dmsimardThe only thing I could think of is to have a link to the build (even though it is still in progress) and then you have a bunch of room to display a bunch of information on an entire page14:39
dmsimardThere's already a lot going on the status page, agreed ;)14:39
tobiashmordred, corvus: so what would be a good and correct wording for 'job waits for node' and 'job waits for parent'?14:40
tobiash'queued' and 'suspended' maybe?14:40
*** logan_ has joined #zuul14:40
tobiashor 'queued' and 'blocked'14:41
dmsimardqueued sounds appropriate but the suspended/blocked wording makes it feel like it's an issue14:41
tobiashqueued and waiting?14:42
dmsimardyou need just one word ? :p14:42
tobiashwell it's one word that must fit into that small box...14:43
mordredwe could just call the status 'dependency'14:43
corvusqueued and waiting sound good.  or we could go with 'node_wait' and 'dep_wait'.14:43
dmsimardcould it just be something on hover ?14:43
mordredcorvus: or those14:43
dmsimardmouse over and you have a small modal or box with more details14:43
dmsimard¯\_(ツ)_/¯14:43
corvus(also, don't forget 'paused')14:43
dmsimardis paused a thing ?14:43
mordreddmsimard: I've been watching status a lot on my phone recently14:43
tobiash'paused' is already there14:44
dmsimardmordred: yeah it works pretty well on mobile14:44
tobiashdep_wait is the most correct I think14:44
corvusyeah, hover is not good ux for something important like this that you want to see at a glance14:44
corvusit should be reserved for extraneous detail that isn't generally useful, or additional information14:45
dmsimardyeah, UX is hard :(14:45
tobiashyes, like the dependency graph14:45
tobiashthat would be useful on hover (or detail link)14:46
corvustobiash: yeah, including the extra bit of information on what it's waiting for is good, and opens up the possibility for more states in the future14:46
corvustobiash: i would be happy to have the job graph rendered right in the box :)14:46
corvusi think that's great visual confirmation of the user's mental model of what the dependency graph is14:47
tobiashyes14:49
tobiashso shall we go with 'node_wait' and 'dep_wait'?14:49
*** rlandy has quit IRC14:49
*** logan- has quit IRC14:49
*** logan_ is now known as logan-14:49
corvustobiash: wfm.  mordred, dmsimard: prefer "node_wait/dep_wait" or "queued/waiting" ?14:55
corvustobiash: yeah, let's go with that for now.  we can always bikeshed in review.  ;)14:57
dmsimardtobiash, corvus: it needs to fit within the grey progress bar right ?14:59
tobiashdmsimard: yes14:59
tobiashcorvus: ok, let's see when I have time to work on this15:00
tobiashbtw, we are running the znode caching in nodepool since 3 weeks in prod now and I didn't see issues related to this15:01
dmsimardcorvus, tobiash: well, we already have "queued" as a status right now so really the one we want to address is the other case ?15:04
mordredcorvus: I like node_wait/dep_wait - I think queued/waiting, while nicer words, don't inherently comunicate the difference to a user who hasn't read the documentation15:04
tobiashmordred: that's right, so while node_wait/dep_wait are more ugly from a visual point of view it's the only proposal so far that communicates the difference clearly to the user15:05
tobiashand most users don't read the docs...15:05
tobiashso I also vote for node_wait/dep_wait15:06
tobiashbtw, what about the queue items that are blocked by semaphore?15:06
tobiashsem_wait on all jobs?15:07
tobiashI lied, semaphores are per job so sem_wait on the job instead of queued is correct too15:07
tobiashso I think we should go with node_wait/dep_wait/sem_wait15:08
mordredtobiash: yeah- I although I wish 'sem' was more naturally clear like node and dep -- but I think having it would be better than nothing15:08
tobiashsemaphore_wait?15:09
mordredif it fits, yeah - I think that would be better15:09
tobiashor mutex_wait knowingly that it's actually a semaphore?15:09
tobiashI don't think it fits15:09
mordredwords are the worst15:09
*** openstackgerrit has joined #zuul15:13
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Fix periodic job display in builds page  https://review.openstack.org/61135215:13
corvusmaybe if the user knows the job uses a semaphore they'll figure out sem_wait15:14
tobiashcorvus: with ^ I think I should abandon my revert?15:14
corvusand.... going back to dmsimard's suggestion, we can *also* use tooltips -- so we can have "sem_wait" with a tooltip of "Waiting for a semaphore currently held by another build"15:15
tobiash(I did it only in case someone wanted a quick fix until you guys are awake)15:15
corvustobiash: yes, i think it's a straightforward fix and i've verified it on production data15:15
tobiash:)15:15
corvusmordred: +3 611352 ?15:15
* tobiash prefers fix forward over revert15:16
*** rlandy_ is now known as rlandy|afk15:16
mordredcorvus: ++15:16
*** bhavikdbavishi has quit IRC15:29
*** samccann has quit IRC15:30
*** samccann_ has joined #zuul15:30
*** bhavikdbavishi has joined #zuul15:30
*** themroc has quit IRC15:46
*** bhavikdbavishi has quit IRC15:46
*** jpena is now known as jpena|off15:47
*** jesusaur has quit IRC16:32
*** panda is now known as panda|off16:36
*** jesusaur has joined #zuul16:38
*** bhavikdbavishi has joined #zuul17:10
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Extract pep8 messages for inline comments  https://review.openstack.org/58963417:14
corvusmordred, Shrews: can you take a look at https://review.openstack.org/610697 ? that will let new users see the build pages in the quick start17:17
Shrewscorvus: sure. tired of company mandated training anyway17:20
mordredcorvus: lgtm17:21
corvustristanC: we don't do anything with the "code: 204" here: http://git.zuul-ci.org/cgit/zuul/tree/zuul/scheduler.py#n1322  --  did you mean to return that as an http status code?  (see http://git.zuul-ci.org/cgit/zuul/tree/zuul/web/__init__.py#n246 where we do handle 404)17:24
corvusmordred: ^ oops i think you wrote that code17:25
corvus(but tristan wrote the original "tenant isn't ready" code)17:26
pabelangerdecided to try quick-start example, but ran into some issue: http://paste.openstack.org/show/732351/ is that know issue before I start looking into it?17:27
corvusturns out it would be really helpful in the quick-start job, but i don't know if it would have any deleterious effects on the web app17:27
pabelangerI am using fedora-28 currently17:27
mordredcorvus: I can't possibly have written buggy code17:27
ShrewsLOLOLOLOLROFLLOLOLOL17:28
pabelangerI am guessing volumes are not working properly17:29
Shrewsi was *totally* lol'ing at something else17:29
corvuspabelanger: thanks for trying it out!  i haven't seen that before, but i haven't been using fedora or sudo.  it's supposed to bind-mount the config files into the container volumes, but it looks like there are permissions problems with that.  i wonder if selinux is involed.17:29
corvuspabelanger: like, maybe selinux doesn't allow the docker daemon to access files in your home directory17:30
pabelangerokay, I don't mind debugging17:30
corvuspabelanger: maybe try cloning zuul to /tmp and running it from there?17:30
pabelangersure17:30
corvuspabelanger: or, of course, setenforce 0 would help confirm (but we should see if we can figure out a way to run this without disabling selinux)17:31
mordredcorvus: I am not sure if it will have negative impacts - but even if it does, I think it's worth working through them17:31
pabelangercorvus: /tmp works, so you might be onto something17:32
pabelangerwill check journald logs and learn more17:32
corvuspabelanger: cool, we should be able to adjust the instructions to just have folks do that17:32
mordredpabelanger, corvus: trying from your homedir with setenforce 0 would also be a nice thing to try - because it also _could_ be just normal unix perms perhaps?17:32
corvusmordred: i'm assuming dockerd runs as root?17:33
corvusif so, unix perms seems unlikely; if not, then yes.  :)17:33
mordredoh - yeah - looks like it- and I thikn we still have the pbrx containers using root too (we should really have them create an unprivileged user)17:34
pabelangermordred: yes, please17:35
mordredpabelanger: it's on the todo-list :)17:35
*** rlandy|afk is now known as rlandy17:39
*** panda|off has quit IRC17:40
*** panda has joined #zuul17:42
pabelangerzuul question, would it be possible for required-projects setting in a job to also take optional path arguement? Like we have today for zuul role in zuul.yaml? There is a use case for ansible-network, that I actually want gorg/foo git repo to be gorg.foo on disk, and with zuul-cloner I was able to do this.17:44
pabelangerfor now I am just doing symlinks17:44
corvuspabelanger: for now a pre-playbook which re-arranges things (or makes symlinks) is probably best; that way you don't have to repeat that in all the jobs.17:51
corvuspabelanger: if we implemented your suggestion, what would you suggest the path be for any implied projects (such as the change under test, or projects which show up in depends-on)?17:52
corvuswe could just ignore the problem, and if the project path of the change under test is important, you could add it to required-projects...17:53
corvusor we could add a new attribute to specify the path of the project under test17:53
corvusi don't see any obvious ways to specify the location of depends-on projects though; they may just have to use the default scheme.17:54
pabelangergood question, I haven't considered depends-on currently17:54
corvusand if we implemented the option, we'd have to get really rigorous about not using hard-coded paths in zuul-jobs17:54
pabelangerYah17:54
corvus(which is not a bad thing, but it is work we'd need to do)17:54
pabelangerthis issue today, from a zuul job, we can control the path to a role onthe executor, with roles setting, but with nested ansible on remote node, cannot do that from zuul job ATM. Figured I'd raise the issue here and discuss.  I think symlinks are okay for now also17:56
corvusyeah, i think we can look into doing that change.  those are just the things off the top of my head that we'd need to consider and also implement.  but i can't think of any major blockers right now.18:04
pabelangerokay, great18:04
*** electrofelix has quit IRC18:14
openstackgerritGonéri Le Bouder proposed openstack-infra/zuul master: encrypt_secret: support OpenSSL 1.1.1  https://review.openstack.org/61141418:44
openstackgerritGonéri Le Bouder proposed openstack-infra/zuul master: encrypt_secret: support OpenSSL 1.1.1  https://review.openstack.org/61141418:46
openstackgerritPaul Belanger proposed openstack-infra/zuul master: WIP: Fedora docker-compose fixes  https://review.openstack.org/61141719:03
pabelangercorvus: mordred: ^ contains fixes for volumes in docker-compose, however I do not know why I also need to install missing packages19:03
*** bhavikdbavishi has quit IRC19:05
pabelangerI need to switch contexts for now, but will look back shortly19:05
*** jesusaur has quit IRC19:08
mordredpabelanger: fascinating19:09
*** jesusaur has joined #zuul19:12
tobiashcorvus: just curious, how is your switch to swift going?19:16
tobiashafter having this in production I have to say it was like a fire and forget switch19:16
tobiashnot a single log storage problem since three months now :)19:17
pabelangertobiash: yah, we too have been happy with swift on vexxhost19:24
pabelangermordred: yah, have no idea right now19:28
pabelangerand hotel booked for Berlin19:28
corvuspabelanger: great, thanks!  the :z flag makes sense, but i don't think we need it on all the volumes, just the ones mounted from the host.20:05
corvuspabelanger: the package thing doesn't make any sense, you should be running the same thing that the quick-start job is running.  that's the whole idea of containers after all.  so let's dig into that some more.  if you can paste the output and the exact image versions used in the executor and setup containers, that would help.20:06
pabelangercorvus: ack, happy to debug more in a bit20:09
corvuspabelanger: (i left comments on the change for when you get back to it)20:09
pabelangerok20:09
jlkJust saw this on the GitHub Universe schedule: https://githubuniverse.com/watch/#migrating-from-gerrit-to-github-enterprise20:15
mordredcorvus: BLAST20:26
mordredcorvus: https://review.openstack.org/#/c/610744/220:26
corvusmordred: hey cool warnings work :)20:41
corvusmordred: and i guess we should have saved the return file20:42
mordredyeah20:43
corvusi think we're just missing an int() i'll fix20:43
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Extract pep8 messages for inline comments  https://review.openstack.org/58963420:45
pabelangercorvus: okay, I tried your suggestions for z flag, left reply with pastebin20:53
corvuspabelanger: okay, i guess :z is required for volumes which are only shared between containers too20:55
pabelangercorvus: I also cannot reproduce the package issue, I did however clean out some old images. I wonder if somehow I was running an old layer some where20:55
corvuspabelanger: that might be it.  so maybe just update the patch to do :z for all the volumes20:55
pabelangersure20:55
pabelangergive me a few minutes to do that20:56
pabelangercorvus: also, I admit it is pretty freaking cool the quickstart docs. Worked perfectly20:56
pabelangergreat work20:56
corvus\o/20:57
corvushas anyone seen the "tenant foo isn't ready" message with the new dashboard?  i ask because i'm running zuul-web and the dashboard locally and i get a javascript error20:57
*** hashar has quit IRC21:01
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Fedora docker-compose fixes for selinux  https://review.openstack.org/61141721:06
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Use zuul/nodepool-launcher container for docker-compose  https://review.openstack.org/61144221:10
pabelangerokay, that should be it for fedora and also noticed container name fix21:10
pabelangerI have to run away and eat with family now21:10
*** jlk has quit IRC21:14
*** jlk has joined #zuul21:16
corvuspabelanger: 2x+2 thanks!21:16
openstackgerritMerged openstack-infra/nodepool master: Use quota handling code for min-ready declines  https://review.openstack.org/61049521:23
openstackgerritMerged openstack-infra/nodepool master: Remove unneeded todo comment  https://review.openstack.org/61049621:24
*** sshnaidm has joined #zuul21:40
*** sshnaidm is now known as sshnaidm|off21:40
*** sshnaidm_ has joined #zuul21:42
*** sshnaidm_ has quit IRC21:42
*** sshnaidm|off has quit IRC21:45
openstackgerritJeremy Stanley proposed openstack-infra/zuul-base-jobs master: Correct zuul-jobs path  https://review.openstack.org/59960721:59
*** rlandy is now known as rlandy|bbl22:31
*** j^2 has quit IRC22:41
mordredcorvus: IT WORKED!!!!!23:03
mordredhttps://review.openstack.org/#/c/610744/ <-- https://review.openstack.org/#/c/610744/2/openstack/cloud/openstackcloud.py23:03
mordredgundalow: ^^ you might also enjoy that23:04

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!