Tuesday, 2019-09-03

*** jamesmcarthur has quit IRC00:06
*** nBorg has quit IRC00:49
*** jamesmcarthur has joined #zuul01:00
*** jamesmcarthur has quit IRC01:17
*** threestrands has joined #zuul01:37
*** jamesmcarthur has joined #zuul02:17
*** jamesmcarthur has quit IRC02:27
*** jank has joined #zuul04:19
*** jamesmcarthur has joined #zuul04:31
*** jamesmcarthur has quit IRC04:35
*** raukadah is now known as chandankumar04:59
*** bhavikdbavishi has joined #zuul05:17
*** fdegir has quit IRC05:40
*** fdegir has joined #zuul05:40
*** nBorg has joined #zuul06:25
*** jpena|off is now known as jpena06:49
*** pcaruana has joined #zuul07:06
*** themroc has joined #zuul07:11
*** threestrands has quit IRC07:18
AJaeger_mhu: via depends-on? That only works for repos that are whitelisted in project-config/zuul/main.yaml for the Zuul tenant.07:19
*** ofosos has joined #zuul07:32
*** gtema has joined #zuul07:58
*** hashar has joined #zuul08:11
*** badboy has joined #zuul08:35
badboyhi guys08:35
badboyI've created a timer trigger pipeline and for some strange reason a script is run for every branch of every repo in Zuul08:36
badboyI just want to run in once per 24 hours not X times REPO_COUNT per 24 hours08:36
mhuthanks AJaeger_ I'll have a look08:39
AJaeger_badboy: time trigger runs by default for each branch. If you have some config to share, we might have some more comments.09:16
*** chandankumar has quit IRC09:18
*** chandankumar has joined #zuul09:19
badboyAJaeger_ I have a script which should be run once per Zuul instance (not for every repo/branch)09:26
badboyAJaeger_it collects all changes pending submission/gating and creates an email09:27
badboyAJaeger_ right now it runs (on my dev instance three times: two branches on one repo, a one branch on another repo)09:28
badboyAJaeger_ funny thing is that it's only bound to one project (repo)09:28
AJaeger_badboy: and you have that configured via periodic timer? That's not how periodic works, it is used to run a job per repo.09:28
badboyAJaeger_ and still runs for the one with only one branch09:28
badboyAJaeger_ http://paste.openstack.org/show/769814/09:32
badboyAJaeger_ pending-changes just executes a script which queries gerrit server and sends an email09:34
AJaeger_badboy: so, yes, the script should run only on first-repo - and one invocation per branch.09:35
AJaeger_You could limit it to master with a "branches:" line09:35
badboyin the pipelines.yaml?09:35
AJaeger_badboy: in first-repo, use "my-job: branches: master"09:36
badboyAJaeger_ like this http://paste.openstack.org/compare/769815/769814/?09:38
AJaeger_badboy: change pending-changes to "pending-changes:", otherwise ok09:43
AJaeger_(indentation of branches might need two more spaces to make yaml happy, please check)09:43
badboyAJaeger_ thank you!09:46
badboyAJaeger_ the resolution: http://paste.openstack.org/compare/769818/769814/09:47
AJaeger_yes - still don't know why it run on the other repo.09:47
* AJaeger_ will be back later...09:47
badboyAJaeger_ my use case is that first-repo has lots and lots of branches but Zuul only covers one therefore I needed to limit the script to only one run09:48
badboyAJaeger_ thanks once more09:48
*** ianw has quit IRC10:00
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Improve job and node information banner  https://review.opendev.org/67797110:06
*** ianw has joined #zuul10:08
*** ianw has quit IRC10:27
*** ianw has joined #zuul10:28
*** ianw has joined #zuul10:29
openstackgerritFabien Boucher proposed zuul/zuul master: Better error handling for the PagureAPIClient  https://review.opendev.org/67974610:33
*** ianw has quit IRC10:34
*** hashar has quit IRC10:35
*** ianw has joined #zuul10:36
*** bhavikdbavishi has quit IRC10:36
*** ianw_ has joined #zuul10:41
*** ianw_ has joined #zuul10:43
*** ianw has quit IRC10:44
*** ianw has joined #zuul10:44
*** tosky has joined #zuul10:45
*** ianw has quit IRC10:46
*** ianw has joined #zuul10:50
*** ianw has quit IRC10:52
*** ianw has joined #zuul10:52
*** gtema has quit IRC11:06
*** hashar has joined #zuul11:10
*** bjackman has joined #zuul11:12
*** jpena is now known as jpena|lunch11:31
*** hashar has joined #zuul11:43
*** badboy has quit IRC12:08
*** bjackman has quit IRC12:09
*** bhavikdbavishi has joined #zuul12:13
*** bogdando has joined #zuul12:16
bogdandohi. I have my local zuul & gerrit & nodepool setup, by CI reproducer scrits, reporting "Project test1 not in pipeline <Pipeline check> for change" . Where to check for wrong configs?..12:16
bogdandoso it drops it from queue w/o triggering job12:17
bogdandozuul.yaml submitted with the change under test http://paste.openstack.org/show/hEHhUycjexbbfDKbPNVy/12:19
pabelangerwhich branch did you use?12:19
pabelanger    branches: ^(?!stable/(newton|ocata|pike)).*$12:19
bogdandomaster12:19
pabelangerpossible you didn't match12:19
bogdandothere is also12:21
bogdando2019-09-03 12:09:19,243 - zuul.Pipeline.tripleo-ci-reproducer.check - DEBUG - [e: 8937218a738843e0ad9afd050ee6fa4b] <QueueItem 0x7fcb8795ec88 for <Change 0x7fcb8a933ac8 test1 1001,2> in check> is a failing item because ['it has an invalid configuration']12:21
bogdandoat the end12:21
pabelangeryah, your vars section doesn't look formatted right12:21
pabelangerbut, I don't think I have tested nested dicts12:21
pabelangerI'm assuming it is okay12:22
pabelangerwhere does tripleo-ci-base-multinode live?12:22
*** gtema_ has joined #zuul12:23
pabelangerbogdando: if you first add12:23
pabelangerhttps://github.com/ansible/ansible-zuul-jobs/blob/master/zuul.d/project-templates.yaml#L79612:23
pabelangerdo your project, and merge, zuul should start to report invalid configurations back to gerrit12:24
pabelangereg: check: jobs: []12:24
openstackgerritBenedikt Löffler proposed zuul/zuul master: Report retried builds in a build set via mqtt.  https://review.opendev.org/63272712:24
openstackgerritBenedikt Löffler proposed zuul/zuul master: Report retried builds via sql reporter.  https://review.opendev.org/63350112:24
bogdandopabelanger: it seems it's zuul.d/tripleo-rdo-base.yaml in zuul-config stored in my local gerrit12:25
bogdandobut that files looks weirdly formatted and has only tripleo-ci-base-multinode-rdo job defined, not tripleo-ci-base-multinode ...12:26
*** bjackman has joined #zuul12:26
pabelangeryah, you need to have zuul report back errors, will explain what the issue is with configuration12:27
pabelangereven if you add noop-jobs first12:28
bogdandopabelanger: thanks! will try that12:28
*** weshay has joined #zuul12:34
*** jpena|lunch is now known as jpena12:40
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Switch PDF file name to doc-PROJECT.pdf  https://review.opendev.org/67977712:43
openstackgerritFabien Boucher proposed zuul/zuul master: Better error handling for the PagureAPIClient  https://review.opendev.org/67974612:44
*** jamesmcarthur has joined #zuul12:46
*** bjackman has quit IRC12:51
*** dmsimard has quit IRC12:56
*** sanjayu_ has quit IRC12:58
*** dmsimard has joined #zuul12:58
*** bogdando has left #zuul12:59
*** sanjayu_ has joined #zuul13:02
*** pcaruana has quit IRC13:03
*** pcaruana has joined #zuul13:34
*** pcaruana has quit IRC13:47
*** pcaruana has joined #zuul13:57
*** bhavikdbavishi has quit IRC14:05
*** pcaruana has quit IRC14:07
*** pcaruana has joined #zuul14:10
*** sanjayu_ has quit IRC14:23
*** michael-beaver has joined #zuul14:31
*** bjackman has joined #zuul14:44
*** bjackman has quit IRC14:50
clarkbpabelanger: thinking about https://review.opendev.org/#/c/679145/2 and what you said about ceph swift yesterday. We might want to do a coordinated update between my chnage there and an update to the swift logs role to add a per install prefix/suffix to container names to help make them globally unique?14:54
clarkbpabelanger: can you confirm that you cannot create containers in vexxost if opendev has already created a container with that name?14:55
pabelangerclarkb: sure, need a container name14:56
*** jank has quit IRC14:57
clarkbpabelanger: logs_00 through logs_09 in ca-ymq-1 exist14:58
pabelangerhttp://paste.openstack.org/show/770138/14:59
pabelangerno work15:00
clarkbneat15:00
clarkbok I'll work on that change after meetings and figure out testing15:01
clarkbhrm can't use hostname because we can have different executors (or maybe that is good enough)15:04
clarkbmay need ot think about the best way to approach this15:04
*** gtema_ has quit IRC15:18
*** hashar has quit IRC15:21
*** pcaruana has quit IRC15:27
clarkbI think what we need is a zuul installation identifier that we can add as a container name suffix15:28
clarkb(or prefix)15:28
clarkbbut I'm not seeing anything like that in place already15:28
clarkbother options: use the executor hostname (then we'll just get more containers), maybe hash it15:29
clarkbset the site identifer as a role var and users can set it if using an object store that requires globally unique container names15:30
clarkbthis last option is simpliest to implement15:31
clarkband would be backward compatible with adding functionality to zuul itself to have a site identifier (we could set the default var value to that site identifier)15:32
*** chandankumar is now known as raukadah15:32
*** bjackman has joined #zuul15:33
*** themroc has quit IRC15:37
*** tosky_ has joined #zuul15:40
*** tosky has quit IRC15:41
*** tosky_ is now known as tosky15:41
*** bjackman has quit IRC15:42
*** bjackman has joined #zuul15:43
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Document globally unique container name requirement  https://review.opendev.org/67981415:47
*** bjackman has quit IRC15:47
clarkbpabelanger: ^ ok I think that is the simplest thing for now. The role supports dealing with this but does require some configuration (so add docs for that)15:47
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths  https://review.opendev.org/67914516:00
clarkbnow to see if I ansibled properly for testing that other related change16:00
*** sgw has quit IRC16:04
*** noorul has joined #zuul16:08
*** jpena is now known as jpena|off16:13
noorulHi16:21
noorulCan someone help me to setup ARA reporting?16:21
dmsimardnoorul: what's up ?16:22
nooruldmsimard: Looking for a document to integrate ARA with Zuul16:23
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths  https://review.opendev.org/67914516:24
dmsimardnoorul: I don't think there's a whole lot of docs on that topic but we can improve that. Are you using the ara-report role ?16:25
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths  https://review.opendev.org/67914516:26
nooruldmsimard: https://etherpad.openstack.org/p/rWL36RmF6W line 16816:27
*** hashar has joined #zuul16:28
pabelangernoorul: https://github.com/ansible/project-config/blob/master/zuul.d/jobs.yaml#L37 are the values we setup for ara-report, note ara_report_executable needs to point to the copy inside the executor venv16:30
noorulpabelanger: I am looking for steps to setup ARA16:31
pabelangerit should be installed by default, IIRC16:31
dmsimardnoorul: unless mistaken, ara is installed by default in the virtualenvs alongside ansible16:31
pabelangerisn't it part of zuul ansible setup now?16:31
pabelangerhttps://opendev.org/zuul/zuul/src/branch/master/zuul/lib/ansible-config.conf16:32
pabelangeryah16:32
pabelangerso, should be inside ansible venv, on executor16:32
pabelangeryou likely just need to setup ara path16:32
noorulI see multiple ansible version folders under /var/lib/zuul16:33
pabelangeryup, that is all the version of ansible, that zuul jobs will be able to use16:33
nooruls/version/versions16:33
noorulI don't see any virtualenv under them16:34
dmsimardnoorul: does ara-report fail ? are you seeing any errors ?16:34
dmsimardnoorul: the virtualenvs are set up inside the workspace at the beginning of the job16:34
nooruldmsimard: So every time ara is installed in the virtual env?16:35
nooruldmsimard: I don't see /opt/venv folder also16:35
pabelanger/opt/venv is site specific, we do that for zuul.a.c16:37
dmsimardnoorul: /opt/venv is probably from pabelanger's environment16:37
dmsimardnoorul: the virtualenv dependencies are in https://opendev.org/zuul/zuul/src/branch/master/zuul/lib/ansible-config.conf16:37
pabelangeryou can just switch that to /var/lib/zuul/16:37
dmsimardbrb in a bit, lunch16:38
noorulWhich is the default location?16:39
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths  https://review.opendev.org/67914516:39
*** brennen|afk is now known as brennen16:40
pabelanger/var/lib/zuul/ansible/ I think16:40
clarkbnoorul: pabelanger the default depends on how you run the installs :/16:41
clarkb/usr/lib/zuul/ansible if you run the manage venvs command16:41
clarkbif however you let zuul install at startup it will go to /var/lib/zuul/ansible-bin iirc16:42
pabelangeroh, that is odd16:42
noorulFollowed https://zuul-ci.org/docs/zuul/admin/zuul-from-scratch.html16:42
noorulhttps://zuul-ci.org/docs/zuul/admin/zuul_install.html16:43
noorulI think I ran sudo zuul-manage-ansible16:43
clarkbnoorul: the zuul-manage-ansible command will install to /usr/lib/zuul/ansible by default16:43
noorulclarkb: cool, I see that16:44
noorulclarkb: Also the log files are getting compressed by default. Is there an option to disable that?16:49
clarkbnoorul: that will depend on how you are collecting the logs I think.16:50
clarkbdo you know what role(s) you are using forthat?16:50
pabelangernoorul: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-October/000565.html might be of interest16:51
noorulLine 205 https://etherpad.openstack.org/p/rWL36RmF6W16:51
pabelangernoorul: yah, you'll want something like https://review.opendev.org/567696/16:52
noorulpabelanger: It recommends compressing16:52
noorulpabelanger: If I compress then browser cannot display that inline16:53
pabelangerright, that is what my ML post is about16:53
pabelangeryou need to update log server to support gzip16:53
pabelangerwhich is an extra step16:53
pabelangerand IMO, day 2 operation16:53
noorulpabelanger: I have gzip enabled on nginx16:54
noorulpabelanger: I think I am missing something on nginx side16:55
noorulpabelanger: Is that PR a WIP?16:55
pabelangeryes, you need to decompress .gz files and browser will load them16:57
noorulpabelanger: In the ML thread, clarkb proposed to improve the doc. Is it available now?16:58
noorulpabelanger: I mean example nginx config16:58
clarkbI don't remember offering to improve nginx docs17:00
clarkbI pushed changes for a bunch of other things though that merged17:00
noorulIs anyone familiar with configuring nginx to decompress?17:01
pabelangeryou need to add a new location, wild card for log.gz files, then gzip off, and add type of text/plain, IIRC17:05
noorulpabelanger: I will try that17:05
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths  https://review.opendev.org/67914517:05
noorulFor ARA after setting the path, ara reporting is still failing17:06
dmsimardnoorul: what is failing ? What is the error ?17:06
nooruldmsimard: I think there is a mistake in my config17:07
nooruldmsimard: Let me try fixing it17:08
*** hashar has quit IRC17:13
nooruldmsimard: It is looking for /usr/lib/zuul/ansible/2.7.13/bin/ara17:15
clarkbbut ansible/2.7/bin/ara is what exists on disk?17:16
nooruldmsimard: On my system it is here /usr/lib/zuul/ansible/2.7/bin17:16
clarkbya so that is a bug in the role I think17:16
clarkbor a mismatch in expectations between how zuul installs ansible and ara and the ara role17:16
noorul{{ ansible_version.full }}17:18
noorul{{ ansible_version.major }}.{{ ansible_version.minor }}17:18
noorulwill that work?17:18
clarkbcorvus: pabelanger https://review.opendev.org/#/c/679145/ should be ready for rereview now. It is tested17:19
clarkbnoorul: yes I think so17:19
*** mattw4 has joined #zuul17:21
AJaeger_could I get another review for a zuul-jobs change, please? https://review.opendev.org/67977717:22
*** hogepodge has left #zuul17:24
noorulhe he17:26
noorulat last ARA report is generated17:26
*** mattw4 has quit IRC17:29
*** pcaruana has joined #zuul17:29
*** hashar has joined #zuul17:30
nooruldmsimard clarkb pabelanger Thank you!17:31
*** noorul has quit IRC17:33
*** mattw4 has joined #zuul17:34
*** jamesmcarthur has quit IRC18:19
*** tosky has quit IRC18:24
*** jamesmcarthur has joined #zuul18:47
*** igordc has joined #zuul19:02
*** AJaeger_ is now known as AJaeger19:03
*** jamesmcarthur has quit IRC19:04
*** jamesmcarthur has joined #zuul19:20
*** sgw has joined #zuul19:37
*** michael-beaver has quit IRC19:40
*** hashar has quit IRC19:50
*** pcaruana has quit IRC20:00
*** jamesmcarthur has quit IRC20:27
*** jamesmcarthur has joined #zuul20:30
openstackgerritMerged zuul/zuul-jobs master: Switch PDF file name to doc-PROJECT.pdf  https://review.opendev.org/67977720:43
pabelangerso, looking at https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L9 I can't figure out how to make that a little more redundant, for example. If  upload-logs-swift fails on the random choice, for what ever reason, POST_FAILUREs will be the result20:53
*** jamesmcarthur has quit IRC20:53
pabelangergiven that include_roles are not able to use until / retries, blocks seem to be the next step20:53
pabelangerhowever, I then cannot figure out how to do more then 2 tries, because of block / rescue logic and nested blocks20:54
pabelangerI don't think you can use a block in a rescue20:54
pabelangerthe only other option I can thing of, is passing all the zuul_log_cloud_config into upload-logs-swift then, adding the retries to the task level20:55
pabelangerwhich is a change of functionality, but maybe best option20:55
pabelangeradding it to https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-swift/tasks/main.yaml#L1920:56
pabelangerwith_random_choice / x times / sleep x seconds20:56
clarkbpabelanger: are you specifically wanting to retry with a different cloud?20:58
pabelangeryah20:58
pabelangerI can retry with same cloud, but that is still a single point of failure20:59
clarkbya so even if you could retry until you'd need some way to remove the failing cloud from the random list21:01
pabelangeryah, for start was just to random n+1 of list size21:01
pabelangerbut could be unlucking and hit same provider multiple times in row21:01
clarkbyup21:02
pabelangerbut, we could pop item off list21:02
pabelangerand reassign21:02
clarkbare the failures you are experiencing not expected to go away on a subsequent retry?21:02
clarkb(just thinking that retrying the same cloud is super easy)21:03
pabelangerno, I'd say it would be infra issue. For some reason an outage21:04
pabelangerso, really would want to use another cloud in that use case21:04
*** nBorg has quit IRC21:51
fungipaladox just shared this in #openstack-infra but it's pretty on-topic in here too: https://bugs.chromium.org/p/gerrit/issues/detail?id=1141822:05
paladoxDoes zuul support http only (that being ssh being optional?)22:05
clarkbpaladox: I think if you only trigger periodically22:06
paladoxoh22:06
clarkbit can report back via http only (this is how opendev's zuul does it) but it relies on the ssh event stream to get events22:06
fungiyeah, it needs ssh access for the gerrit event stream22:07
fungiif gerrit implements a non-ssh event stream i'm quite sure zuul would happily add support for it22:07
paladoxah ok. I doin't think gerrit-review will be able to use that part (google disabled ssh)22:07
clarkbpaladox: is the new callback method for gerrit implemented yet?22:07
fungii take it they don't expose an event stream for their gerrit then22:07
clarkbif so it probably wouldn't be too had to add support for that to zuul22:07
paladoxclarkb you mean checkers?22:08
clarkbs/had/bad/22:08
clarkbpaladox: yes but specifically the bit that calls back to the checkers22:08
paladoxoh, it's a rest api22:08
fungiare checkers expected to poll gerrit periodically then?22:08
clarkbpaladox: ya there are two ways it can work iirc the first is that checkers poll, the other is checkers register a callbcak endpoint and gerrit calls it on events22:08
paladoxhttps://gerrit.googlesource.com/plugins/checks/+/refs/heads/master/resources/Documentation22:08
clarkbfor zuul the callback method is likely much simpler because zuul wants to be event driven and that is already how the github driver works22:09
clarkbadding polling is not impossible but would be new and different so likely more work22:09
paladoxI'm aware that checkers automatically refreshes it's data in the ui. Not sure about the backend.22:09
fungialso polling would be a lot more laggy22:09
pabelangeris chromium looking to migrate away from jenkins?22:11
paladoxchromium is covered by a seperate team so i'm not sure :(22:12
clarkbhrm those docs seem to be less helpful than the original spec doc22:14
clarkbit isn't clear to me how a checker is supposed to operate from those docs but I seem to remember those two modes of operation with the polling planned to be implemented first22:14
paladoxhttps://github.com/GerritCodeReview/gerrit-ci-scripts/commit/ecce2057a3aad4f92898428814bcca185248c4b322:14
clarkbpaladox: what triggers that?22:16
paladoxif you mean how does thst file get trigger, it gets loaded here https://github.com/GerritCodeReview/gerrit-ci-scripts/blob/91cc091cbeb1f1cb4550c95c403b7bcfda429fcd/jenkins/gerrit-verifier.yaml#L211. If you mean how does it trigger checkers, it doesn't. Checkers collects the results, so the ci is responsible for telling checkers about the results. (that's my understanding). Checkers does support "restart" but not sure how that works with ci..22:18
clarkbya what tells jenkins to run and execute the build that reports back22:18
clarkbthat is the part that zuul needs ssh for today22:19
clarkbadding support for reporting as checkers should also not be bad22:19
paladoxyeh22:20
clarkbanother thing that we might have to reconcilewith checkers is inline comments from CI systems22:20
clarkbI'm going to assume that mordred and corvus learned all about this last week though and can fill us in when they return22:20
paladoxi think that's robot comments22:22
clarkbah ok so submit both a check result and separate robot comments22:23
clarkbthat makes sense22:23
paladoxhttps://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#robot-comment-info22:23
paladoxhttps://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#list-robot-comments22:23
*** mgoddard has quit IRC22:47
*** mgoddard has joined #zuul22:47
*** threestrands has joined #zuul23:26
*** sgw has quit IRC23:44
*** sgw has joined #zuul23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!