| *** panda has quit IRC | 00:01 | |
| *** mattw4 has quit IRC | 00:03 | |
| *** panda has joined #zuul | 00:04 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 00:08 | |
| tristanC | Shrews: is there a bug report somewhere with the k8s issue details? | 00:35 |
|---|---|---|
| tristanC | Shrews: authentication issues may relate to how the zuul executor prepare the kube config, perhaps some provider requires extra settings? | 00:36 |
| corvus | tristanC: i think SpamapS that was encountering that and may have more info | 00:37 |
| corvus | s/that// | 00:37 |
| Shrews | tristanC: you'll have to ask SpamapS | 00:39 |
| tristanC | SpamapS: indeed we do test and use the openshift provider successfully | 00:45 |
| tristanC | pabelanger: there is a pagination widget that may works for the builds page using the skip/limit parameters: https://www.patternfly.org/pattern-library/navigation/pagination/ | 00:49 |
| *** michael-beaver has quit IRC | 01:09 | |
| tristanC | Shrews: SpamapS: or maybe the k8s service account created by nodepool is not working: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/kubernetes/provider.py#L179 | 01:12 |
| *** jamesmcarthur has joined #zuul | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:18 | |
| tristanC | so, according to k8s docs, services account may not work depending on the cluster configuration, see the note at the top of: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ | 01:20 |
| tristanC | it's unclear what type of behavior are required, but it seems like we should requires https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#serviceaccount | 01:21 |
| tristanC | and some rbac according to: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#upgrading-from-1-5 | 01:23 |
| tristanC | for example, gke doesn't seems to authorize service account to be used from the outside of the cluster | 01:27 |
| *** sanjayu_ has quit IRC | 01:31 | |
| pabelanger | Shrews: thanks for thre review, didn't see it before | 01:32 |
| pabelanger | tristanC: okay, thanks. I'll try web 2.0 skills :) | 01:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: model: add annotateLogger procedure https://review.opendev.org/663819 | 02:07 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:12 | |
| *** rlandy|ruck has quit IRC | 02:15 | |
| *** sean-k-mooney has quit IRC | 02:28 | |
| *** jamesmcarthur has joined #zuul | 02:29 | |
| *** threestrands has joined #zuul | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:58 | |
| *** sean-k-mooney has joined #zuul | 03:00 | |
| *** sean-k-mooney has quit IRC | 03:14 | |
| *** sean-k-mooney has joined #zuul | 03:16 | |
| *** threestrands has quit IRC | 04:12 | |
| *** sean-k-mooney has quit IRC | 04:19 | |
| *** sean-k-mooney has joined #zuul | 04:21 | |
| *** swest has joined #zuul | 04:43 | |
| *** raukadah is now known as chandankumar | 04:55 | |
| *** flepied has joined #zuul | 05:45 | |
| *** gtema_ has joined #zuul | 06:05 | |
| *** evgenyl has quit IRC | 06:20 | |
| *** evgenyl has joined #zuul | 06:24 | |
| *** sanjayu_ has joined #zuul | 06:27 | |
| *** sanjayu_ has quit IRC | 06:32 | |
| *** pcaruana has joined #zuul | 06:35 | |
| *** yolanda__ is now known as yolanda | 06:56 | |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul-jobs master: Revert "Explicitly store date facts for promote" https://review.opendev.org/663848 | 06:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: model: add annotateLogger procedure https://review.opendev.org/663819 | 06:58 |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul-jobs master: Revert "Explicitly store date facts for promote" https://review.opendev.org/663848 | 06:58 |
| *** hashar has joined #zuul | 07:14 | |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul-jobs master: Revert "Explicitly store date facts for promote" https://review.opendev.org/663848 | 07:21 |
| *** jpena|off is now known as jpena | 07:53 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Explicitly store date facts for promote" https://review.opendev.org/663848 | 07:58 |
| *** sanjayu_ has joined #zuul | 08:00 | |
| *** sanjayu_ has quit IRC | 08:09 | |
| *** saneax has joined #zuul | 08:10 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-jobs master: install-nodejs: add support for RPM-based OSes https://review.opendev.org/631049 | 08:49 |
| ofosos | Is there anything I need for the test to run despite a zookeeper daemon? I think my tests fail while waiting for a node to run jobs on.... :( | 08:51 |
| ofosos | (Want to run locally) | 08:53 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Add Bitbucket Server source functionality https://review.opendev.org/657837 | 08:54 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter https://review.opendev.org/658335 | 08:54 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 08:54 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Upgrade formatting of the patch series. https://review.opendev.org/660683 | 08:54 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 08:54 |
| *** zbr has joined #zuul | 09:29 | |
| *** sanjayu_ has joined #zuul | 09:34 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:35 |
| *** saneax has quit IRC | 09:36 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:51 |
| *** flepied has quit IRC | 10:00 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:13 |
| *** gtema has joined #zuul | 10:34 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:35 |
| *** gtema_ has quit IRC | 10:35 | |
| *** zbr is now known as zbr|rover | 10:44 | |
| *** flepied has joined #zuul | 10:50 | |
| openstackgerrit | Aurelio Jargas proposed zuul/zuul master: Break long repo names to make them fit https://review.opendev.org/663899 | 11:12 |
| *** sanjayu_ has quit IRC | 11:19 | |
| *** saneax has joined #zuul | 11:20 | |
| *** saneax has quit IRC | 11:24 | |
| *** saneax has joined #zuul | 11:25 | |
| *** hashar has quit IRC | 11:31 | |
| *** hashar has joined #zuul | 11:31 | |
| tobiash | ofosos: you mean the tox tests? | 11:31 |
| tobiash | ofosos: in that case you could just run the tools/test-setup-docker.sh script that starts all test requirements in containers | 11:32 |
| *** jpena is now known as jpena|lunch | 11:38 | |
| ofosos | tobiash: thanks, that sounds good :) | 11:38 |
| ofosos | Does the `tenant_config_script' get executed just once during startup? | 11:41 |
| *** EmilienM is now known as EvilienM | 11:45 | |
| *** [GNU] has joined #zuul | 12:03 | |
| [GNU] | hey all, is anyone aware of a job that is waiting for a pod to be terminated successful, so that I can peek at the playbook and learn from it!? :) | 12:04 |
| *** sanjayu_ has joined #zuul | 12:05 | |
| [GNU] | kubernetes pod ... btw :) | 12:07 |
| *** saneax has quit IRC | 12:08 | |
| *** sanjayu__ has joined #zuul | 12:08 | |
| *** sanjayu_ has quit IRC | 12:11 | |
| *** sanjayu_ has joined #zuul | 12:19 | |
| *** sanjayu__ has quit IRC | 12:22 | |
| *** ianychoi_ has joined #zuul | 12:23 | |
| *** ianychoi has quit IRC | 12:27 | |
| *** paladox has quit IRC | 12:27 | |
| *** flaper87 has quit IRC | 12:27 | |
| *** flaper87 has joined #zuul | 12:29 | |
| *** rlandy has joined #zuul | 12:32 | |
| *** rlandy is now known as rlandy|ruck | 12:33 | |
| *** jpena|lunch is now known as jpena | 12:47 | |
| *** sanjayu_ has quit IRC | 13:07 | |
| *** saneax has joined #zuul | 13:12 | |
| *** felixgcb has joined #zuul | 13:20 | |
| *** jamesmcarthur has joined #zuul | 13:32 | |
| *** chandankumar is now known as raukadah | 14:06 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:29 |
| felixgcb | Hey :) is somebody already using zuul's nodepool ontop of kubernetes (or aws eks)? I wonder if there is some config repo / blog post I could check out... | 14:35 |
| clarkb | felixgcb: I think tristanC is using it with openshift. I don't know if they have written any blog posts about it | 14:40 |
| clarkb | that is the software factory zuul install | 14:40 |
| tobiash | ofosos: it gets executed during startup and full reconfigurations | 14:42 |
| [GNU] | felixgcb, on OpenShift?! :) yes | 14:42 |
| [GNU] | tristanC, has help a lot with that and has some good blogs on it | 14:43 |
| *** paladox_ has joined #zuul | 14:51 | |
| *** paladox_ is now known as paladox | 14:53 | |
| *** paladox is now known as paladox__ | 14:53 | |
| *** paladox__ is now known as paladox | 14:53 | |
| felixgcb | [GNU]: nah on AWS EKS, but I heard the openshift driver can also be used for that. Do you have the links for those blogposts by any chance? | 14:56 |
| felixgcb | I think the kubernetes driver would be more fitting, but it says in the docs that it needs cluster admin permissions. That's a little much :D | 14:58 |
| [GNU] | felixgcb, ja, the openshift driver is basically just using kubectl (iirc), pls have a look around https://www.softwarefactory-project.io/category/blog.html | 14:58 |
| [GNU] | hmm, I'm not a cluster admin with the openshift driver | 14:59 |
| felixgcb | Yes the openshift driver needs much less permissions | 14:59 |
| felixgcb | Ok I'll have a look :) | 14:59 |
| clarkb | I think with k8s you probably don't want to share that cluster with zuul and other things? especially if testing untrusted code | 15:00 |
| clarkb | openshift has a lot more isolation built in that makes that safer aiui | 15:00 |
| Shrews | felixgcb: easystack uses k8s, but they don't seem to participate in this channel, sadly. their last talk is here: https://www.youtube.com/watch?v=3AHRFORLLFE | 15:00 |
| [GNU] | clarkb, +1 +1 | 15:00 |
| clarkb | so giving out cluster admin to zuul isn't a huge deal as it should be a dedicated cluster anyway | 15:00 |
| ofosos | tobiash, cool | 15:01 |
| felixgcb | clarkb: Hm I see that zuul calls these projects "untrusted projects", but in our case this will be company-internal projects written by our devs, not some codesnippets from the www | 15:02 |
| felixgcb | Shrews: Ah cool, I'll check out their talk :) thanks! | 15:02 |
| clarkb | felixgcb: I would advocate that you not treat unreviewed code much differently regardless of it coming from the internet or intenally. We all make mistakes and this gives you a nice layer of safety | 15:03 |
| felixgcb | clarkb: Hm that's an interesting take on unreviewed code... | 15:04 |
| *** gtema has quit IRC | 15:15 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 15:16 |
| *** hashar has quit IRC | 15:17 | |
| clarkb | felixgcb: part of the job of the machines is to help ensure these changes are safe. We lack that information while running the checks | 15:19 |
| *** saneax has quit IRC | 15:20 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 15:49 |
| fungi | felixgcb: also, using "this will be company-internal projects written by our devs" as a reason to treat all repos as trusted implies you're okay with random internal devs compromising and/or bypassing the ci system at any time | 15:56 |
| fungi | it would be like saying you don't want to put passwords on your office network equipment because only employees connect through it | 15:56 |
| *** michael-beaver has joined #zuul | 15:57 | |
| *** mattw4 has joined #zuul | 16:07 | |
| *** panda is now known as panda|off | 16:13 | |
| corvus | fbo: i think the pagure driver is ready to merge after you fix up https://review.opendev.org/661316 and rebase, yeah? | 16:19 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP add repl https://review.opendev.org/579962 | 16:24 |
| *** flepied has quit IRC | 16:25 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: [DNM] Test unittests and multinode with base-test https://review.opendev.org/663995 | 16:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Proposed spec: tenant-scoped admin web API https://review.opendev.org/562321 | 16:56 |
| *** jpena is now known as jpena|off | 16:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add opendev tarball jobs https://review.opendev.org/664006 | 17:12 |
| *** felixgcb has quit IRC | 17:24 | |
| mattw4 | Hi Zuul team: can anyone tell me if I can tell Zuul to gather "full" ansible_facts from the test node, rather than the minimum? I'm missing ansible_interfaces, required by devstack setup in the minimum ansible_facts. | 17:53 |
| mattw4 | i.e. gathering the minimum facts with ansible "ansible all -m setup -a 'gather_subset=!all'" produces output similar to that gathered by Zuul executor | 17:54 |
| clarkb | mattw4: the playbook can have a setup task to get them all iirc | 17:54 |
| *** jamesmcarthur has quit IRC | 17:54 | |
| clarkb | I want to say that is how it works for the opendev install. We do a smaller amount for most jobs to speed things up then jobs that need more opt into more | 17:55 |
| mattw4 | clarkb: I can just instruct the playbook to run the 'full' ansible_facts setup? I will see if I can figure that out. | 17:55 |
| clarkb | yes you should be able to | 17:56 |
| clarkb | with a setup task | 17:56 |
| pabelanger | Yup, that is correct | 17:59 |
| mattw4 | thanks clarkb & pabelanger, testing now... | 18:06 |
| *** gtema has joined #zuul | 18:11 | |
| mattw4 | clarkb, pabelanger: thanks again. Added a "setup" role to the base pre.yaml and that provided the full set of ansible_facts! | 18:24 |
| gtema | clarkb: I think we need to drop image checksum validation in the nodepool image upload, since after I really repaired it I again got confirmation it is not reliable | 18:25 |
| gtema | tests in SDK are failing now and you can really see from the nodepool logs that the checksum returned is not the one we expect | 18:26 |
| clarkb | gtema: is the issue that glance doesn't calculate the sha as it goes? so there is a lag in reporting the checksum? | 18:26 |
| gtema | see logs under http://logs.openstack.org/38/662738/1/check/nodepool-functional-py35-src/f236fc9/controller/logs/screen-nodepool-builder.txt.gz | 18:26 |
| clarkb | gtema: but also if the checksums are different doesn't that mean our images are different? | 18:26 |
| clarkb | and maybe that should be a failure if glance can't write the same image down? | 18:27 |
| gtema | I have no clue how it get's generated, but it is not the one of the image we upload | 18:27 |
| clarkb | gtema: maybe we can add a step to the job that goes and checks the checksum of the file on disk | 18:27 |
| clarkb | that should aid in debugging | 18:27 |
| gtema | this means we need download image | 18:27 |
| clarkb | any file should work I think | 18:28 |
| gtema | and basically in the glance there is some method (I'm not the developer of it, but I know it exists), which does image transformation during upload | 18:28 |
| gtema | meaning checksum should not be really trusted | 18:28 |
| clarkb | we just want to verify the same bits end up on both sides to verify that isn't failing (and that would put the bug in glance's checksum reporting) | 18:28 |
| clarkb | so while such a feature is good for easiers for simplicitly it should also be possible to verify an image as well | 18:29 |
| clarkb | I think we can have both things, but maybe that requires work in glance first | 18:29 |
| gtema | agree | 18:29 |
| gtema | so for the moment I will do only put a fat warning in the log, bug SDK should not fail unless we fix glance | 18:30 |
| fungi | rosmaita or others in #openstack-glance may have suggestions | 18:30 |
| corvus | clarkb, fungi: can you +3 https://review.opendev.org/664006 ? i did a thing there which i think should make that self-testing, even for the promote jobs. that will let us see if all the tarballs.opendev.org stuff is working. | 18:50 |
| clarkb | I can look | 18:50 |
| clarkb | also I should prep for zuul scheduler restart after lunch. Process for that is pull your repl change into local clone, pip3 install that, then restart? | 18:51 |
| corvus | clarkb: t | 18:51 |
| fungi | the nodepool functional failures are expected/known? | 18:52 |
| corvus | clarkb: i think pabelanger would like us to do a full system restart too, to get 2.8 on the executors | 18:52 |
| corvus | fbo: yes | 18:52 |
| corvus | fungi: ^ | 18:52 |
| fungi | are these related to what gtema brought up? | 18:52 |
| fungi | (the image checksum mismatches) | 18:52 |
| corvus | oh, maybe they are not known to me | 18:52 |
| gtema | I do not think. The patch is not yet merged, which enforces checksum verification | 18:53 |
| clarkb | ya it was added wtih a flag you ahve to toggle | 18:53 |
| clarkb | so shouldn't affect nodepool | 18:53 |
| *** armstrongs has joined #zuul | 18:53 | |
| corvus | i think that job just needs to be rewritten | 18:54 |
| fungi | the build-javascript-content job seems to link its results to npm/html which dne | 18:54 |
| corvus | fungi: yep! i didn't write that. | 18:55 |
| fungi | heh | 18:55 |
| fungi | yeah, seems to result in archiving an empty npm directory | 18:55 |
| corvus | hrm, there should be a tarball somewhere | 18:58 |
| corvus | oh, ugh. | 18:59 |
| corvus | that's the wrong job | 18:59 |
| corvus | (which is great, because it means that job does literally nothing) | 18:59 |
| fungi | the build-python-release job does seem to produce artifacts | 19:00 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add opendev tarball jobs https://review.opendev.org/664006 | 19:02 |
| corvus | fungi, clarkb: okay, that's updated with a dependency ^ | 19:02 |
| *** armstrongs has quit IRC | 19:02 | |
| *** gtema has quit IRC | 19:08 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 19:15 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 19:15 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 19:15 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Auto-delete expired autohold requests https://review.opendev.org/663762 | 19:15 |
| openstackgerrit | Merged zuul/zuul master: Break long repo names to make them fit https://review.opendev.org/663899 | 19:16 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add caching of autohold requests https://review.opendev.org/663412 | 19:18 |
| Shrews | sorry for the spam | 19:19 |
| pabelanger | corvus: clarkb: yah, it would be neat if we could do executors too | 19:22 |
| pabelanger | 2.8.1 was released yesterday too | 19:22 |
| *** bhavikdbavishi has joined #zuul | 19:33 | |
| Shrews | corvus: what are the incantations i need in a unit test in test_scheduler in order to trigger the code in _doBuildCompletedEvent(), assuming I'm using @simple_layout('layouts/autohold.yaml') for the test? | 19:43 |
| Shrews | i was hoping a call to self.fake_gerrit.addFakeChange() would be enough, but apparently not | 19:44 |
| corvus | it should be -- that should happen for all builds | 19:45 |
| corvus | Shrews: maybe you could point me at the not-working code and i can poke at it? | 19:45 |
| Shrews | corvus: yeah, 1 sec... trying to test the hold request auto-delete stuff.... | 19:45 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Auto-delete expired autohold requests https://review.opendev.org/663762 | 19:46 |
| Shrews | corvus: the new test in ^^^ | 19:46 |
| Shrews | you may even have a better way to test that | 19:46 |
| Shrews | brb... moar coffee needed | 19:49 |
| corvus | Shrews: ah i see the issue :) | 19:50 |
| corvus | Shrews: left comments | 19:53 |
| Shrews | corvus: ah, thx. it didn't work, so I'm gonna suspect a bug in my code at this point and go from there | 19:57 |
| corvus | Shrews: i haven't traced it all through, but you may need another build run through the system to trigger the expired code path... | 20:03 |
| smcginnis | Back looking at getting updated third party CI instructions written down. Running into an odd one. | 20:03 |
| smcginnis | Got one deployment working, so tried to replicate it on a fresh install to capture more notes. | 20:04 |
| smcginnis | First one would trigger on upstream ci-sandbox changes, new one does not. | 20:04 |
| corvus | smcginnis: the new way is to just make an image and have everyone deploy that in production. ;) | 20:04 |
| smcginnis | I notice a difference in the pipeline event processing, even though the definitions are the same. | 20:04 |
| smcginnis | corvus: That would be nice. :) | 20:05 |
| smcginnis | Here's what I get on the first one that works, followed by the second one that does not: | 20:05 |
| smcginnis | matched <GerritEventFilter types: comment-added ignore_deletes: True event_approvals: Verified:1> in pipeline <IndependentPipelineManager thirdpartyci> | 20:05 |
| smcginnis | does not match <GerritEventFilter types: comment-added ignore_deletes: True event_approvals: Verified:1, username:zuul> in pipeline <IndependentPipelineManager thirdpartyci> because False | 20:05 |
| smcginnis | New one has "event_approvals: Verified:1, username:zuul" but one that works does not mention zuul. | 20:06 |
| smcginnis | But the exact same pipeline definition. | 20:06 |
| smcginnis | And new-comment Verified:1 is coming from the zuul user. | 20:06 |
| clarkb | smcginnis: that looks like a gate pipeline? or are you wanting third party ci to not run until zuul passes it? | 20:06 |
| *** jamesmcarthur has joined #zuul | 20:07 | |
| smcginnis | clarkb: Correct. In Cinder we "recommend" not running until zuul has left +1 to reduce load. Though I think very few actually run that way. | 20:07 |
| pabelanger | IIRC, rdoproject does that | 20:07 |
| clarkb | smcginnis: can you share your pipeline definition? | 20:07 |
| pabelanger | let me find pipeline stanza | 20:08 |
| corvus | smcginnis: i'd start from the assumption that it is somehow not the same pipeline definition and try to figure out why that is. that's a pretty straightforward repr of the filter that zuul creates out of a pipeline trigger. | 20:08 |
| smcginnis | clarkb: http://paste.openstack.org/show/752641/ | 20:08 |
| Shrews | corvus: oh, we process autoholds only on failures, so my req expiration code won't get hit unless a build fails. | 20:08 |
| corvus | or, if you don't really care that they're different and just want the second one to work, we can ignore that and go from there. | 20:08 |
| smcginnis | corvus: I guess I don't care too much, just want it to work, but I can do another stare and compare to see if somehow there was a copy paste error. | 20:09 |
| Shrews | corvus: not sure how i feel about that, but that's really the only place to do that check currently | 20:09 |
| corvus | Shrews: self.executor_server.failJob can help make a failed job | 20:09 |
| corvus | smcginnis: let's just roll forward then :) | 20:10 |
| smcginnis | It feels like it has to be really close, so I will not be surprised if there is one character missing in a yaml file or something. | 20:11 |
| *** jamesmcarthur has quit IRC | 20:12 | |
| clarkb | need to dedent the username | 20:12 |
| corvus | smcginnis: can you paste all of your scheduler logs from the time the gerrit event is received until it's done with it? | 20:12 |
| clarkb | username is a field of the comment not the vote (at least how zuul is processing it) | 20:13 |
| smcginnis | Getting fresh logs... | 20:13 |
| corvus | clarkb: username is valid in either location | 20:14 |
| clarkb | corvus: even in a trigger? I see we do it that way for requires | 20:14 |
| smcginnis | I couldn't find a detailed guide on options for that pipeline definition, so if there is a better way to define it so it is only triggered on zuul +1, I'm open to any suggestions. | 20:14 |
| clarkb | (I'm guessing it has to do with the structure of the trigger event from gerrit) | 20:14 |
| corvus | clarkb: yeah, we have it that way in our trigger | 20:15 |
| smcginnis | Someone added this long ago: https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#How_do_I_trigger_my_CI_to_rerun_on_gerrit_comments.3F | 20:15 |
| smcginnis | THat actually used require-approval, but based on last convo here that sounded wrong. | 20:15 |
| corvus | smcginnis: we have very detailed docs -- https://zuul-ci.org/docs/zuul/admin/drivers/gerrit.html#trigger-configuration | 20:15 |
| pabelanger | smcginnis: looks like we never migrated it to zuulv3, but https://github.com/rdo-infra/review.rdoproject.org-config/blob/eed6606ab1436012d89cbd9a81a9abb34290373c/zuul/upstream.yaml#L22 was v2 version of openstack-check-verified. | 20:16 |
| pabelanger | maybe be some help | 20:16 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 20:16 |
| Shrews | corvus: got it. thx for the help | 20:16 |
| smcginnis | pabelanger: Thanks, maybe that does help. | 20:16 |
| smcginnis | Should the username be under approval or at the same level as event and approval? | 20:17 |
| corvus | clarkb: oh, our username is on require approval, not approval | 20:17 |
| pabelanger | smcginnis: and where we deleted, with new zuul user name | 20:17 |
| smcginnis | In mine, https://github.com/rdo-infra/review.rdoproject.org-config/blob/eed6606ab1436012d89cbd9a81a9abb34290373c/zuul/upstream.yaml#L37 is indented | 20:17 |
| pabelanger | https://github.com/rdo-infra/review.rdoproject.org-config/commit/4f0733e0b7cd71d7d094d24fba26af20f275f8fb#diff-56448f4b4b65ff4e7b61a23290de1b96L34 | 20:17 |
| corvus | clarkb: i think you might be right | 20:18 |
| clarkb | ya and we document it that way too | 20:18 |
| *** bhavikdbavishi has quit IRC | 20:19 | |
| pabelanger | yah, line 72 in pb looks indented 1 too many | 20:19 |
| corvus | smcginnis: ^ so yeah, assuming you want to stick with 'approval' rather than 'require-approval' (which i think you do), you should dedent the username | 20:19 |
| smcginnis | corvus: Cool, test running right now. | 20:19 |
| smcginnis | corvus, clarkb, pabelanger: Thank you! That worked. | 20:20 |
| smcginnis | I still don't understand how my first setup was working right, but I'm not going to dwell on that right now. | 20:20 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 20:22 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 20:22 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 20:22 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 20:41 |
| Shrews | mordred: that WIP ^^ is the last puzzle piece | 20:41 |
| Shrews | and quite the puzzle it was | 20:42 |
| Shrews | though there is also the integration with the web admin spec stuff, but not worried about that right now | 20:47 |
| evgenyl | A quick question on nodepool, when I configure my openstack provider, can I use the images that are already present in glance, instead of having a builder process doing the build and uploading the images? | 21:11 |
| clarkb | evgenyl: yes you can specify images to use instead | 21:11 |
| evgenyl | clarkb: Do I need a builder process running in this case? | 21:12 |
| clarkb | evgenyl: https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].cloud-images | 21:12 |
| clarkb | no you won't need a builder in that case | 21:12 |
| evgenyl | clarkb: This is awesome, thank you! | 21:13 |
| mattw4 | Hi Zuul team: this might not be the right forum, but can anyone tell me which role is supposed to clone devstack before role:orchestrate-devstack runs? | 21:19 |
| mnaser | mattw4: zuul itself clones devstack afaik | 21:20 |
| mnaser | because it's listed as required-projects: in the job definition | 21:20 |
| fungi | evgenyl: you will, however, need to be able to provide access credentials for zuul (rather ansible) to be able to authenticate to the instances booted from those images | 21:20 |
| clarkb | mnaser: correct, part of zuul's job is putting all the repos a job needs on disk in the test env | 21:20 |
| mnaser | i'm speaking about the general use case anyways | 21:20 |
| clarkb | fungi: nodepool too | 21:20 |
| mattw4 | mnaser, clarkb : do I need move devstack into /opt/stack/devstack with another role? | 21:21 |
| SpamapS | IIRC the role that pushes it from zuul-executor -> nodes is prepare-workspace | 21:21 |
| SpamapS | mattw4:why not just use it where it is, {{ zuul.projects['openstack/devstack'].src_dir }} | 21:21 |
| clarkb | I think the zuulv3 native jobs do run devstack in place | 21:22 |
| clarkb | only the legacy jobs do the moving around | 21:22 |
| mattw4 | SpamapS, I figured that was where it would end up, but the run-devstack role seems to expect it there | 21:22 |
| fungi | clarkb: doesn't the job (or one of its parents, for example the base job) need the mirror-workspace-git-repos role to take care of getting that repo (and others) onto the remote node? | 21:22 |
| clarkb | fungi: its the one SpamapS pointed out | 21:23 |
| clarkb | fungi: iirc its documented as part of base job set up | 21:23 |
| fungi | ahh, yeah, we were typing it simultaneously it seems | 21:23 |
| evgenyl | fungi: Will specifying `key-name` in nodepool.yaml and matching `executor.private_key_file` in zuul.conf be sufficient to provide the access? | 21:23 |
| fungi | evgenyl: as long as your cloud provider exposes those keys in such a way that the booted instances can consume them (for example from a metadata server or configdrive record) | 21:24 |
| evgenyl | fungi: I see, thank you. | 21:24 |
| clarkb | mattw4: you are right that devstack_base_dir seems to default to /opt/stack | 21:24 |
| fungi | evgenyl: so if your images have cloud-init or glean or something like that preinstalled then should work fine | 21:25 |
| clarkb | so maybe that was kept in the job conversion? | 21:25 |
| clarkb | thankfully we can look at jobs that have run to get more info | 21:25 |
| mattw4 | clarkb: yeah, I figured Zuul would move devstack into the right place, but maybe I've misconfigured a variable? | 21:26 |
| fungi | clarkb: i think there's a role in devstack which copies the repos from the zuul-provided copies into the stack user home | 21:26 |
| mattw4 | fungi: that's what I was expecting | 21:26 |
| clarkb | fungi: wow so we copy from /opt/ to /home/ then back again amazing | 21:26 |
| fungi | hah | 21:26 |
| clarkb | mattw4: look in devstack's pre.yaml | 21:27 |
| clarkb | setup-devstack-source-dirs is a role called there | 21:27 |
| clarkb | ya that does it | 21:27 |
| clarkb | no wonder these jobs are so slow on rax nodes | 21:27 |
| clarkb | (/opt is a separate device there) | 21:28 |
| fungi | yeah, since /opt and /home are on separate devices there | 21:28 |
| fungi | less of an issue on providers with a coalesced rootfs | 21:28 |
| fungi | since the data isn't shuffled around, just a bunch of hardlinks added | 21:28 |
| mattw4 | clarkb: I've got devstack source in /root/src/opendev.org/openstack/devstack, but not in /opt/stack/devstack. Wonder what I did :/ | 21:28 |
| clarkb | mattw4: looks like your jobs use the root user (not zuul user) and you aren't running the devstack pre.yaml? | 21:29 |
| clarkb | or the find in that setup playbook isn't finding the files due to the paths (though they are relative so that should work despite the path chagne I think) | 21:30 |
| mattw4 | clarkb: most of the Ansible seems to be as root, where can I specify that the job user as stack? | 21:30 |
| clarkb | mattw4: in our case we run the jobs as the 'zuul' user | 21:30 |
| mattw4 | clarkb, I think it's the setup playbook not finding them | 21:30 |
| clarkb | mattw4: that is a setting of nodepool and zuul I think | 21:30 |
| clarkb | nodepool sets that and passes it to zuul | 21:31 |
| mattw4 | clarkb: yeah, I should specify the zuul user in nodepool | 21:31 |
| clarkb | https://zuul-ci.org/docs/nodepool/configuration.html#attr-diskimages.username | 21:31 |
| clarkb | if it is a file path issue that may fix it. Though I don't think it is a filepath issue. Can you look in your logs for when that role runs to see what it is doing? | 21:32 |
| mattw4 | clarkb: here's the setup-devstack-dirs role running under devstack/pre.yaml: http://paste.openstack.org/show/752643/ | 21:34 |
| mattw4 | so I have all the project dirs including nova, neutron, placement, swift, etc. in /opt/stack, but no devstack there :/ | 21:36 |
| *** rlandy|ruck has quit IRC | 21:37 | |
| clarkb | mattw4: you might need to add devstack as a required project if it isn't already? | 21:38 |
| mattw4 | clarkb, that makes sense - lemme check | 21:38 |
| mattw4 | clarkb, it's a required project in the job definition. Should that be a list of all the projects that the devstack job will setup? | 21:40 |
| clarkb | ya its all the source repos devstack needs to function | 21:41 |
| clarkb | maybe check permissions? could be the copy is failing due to that? | 21:45 |
| fungi | i would hope the copy failing would show up in the console log | 21:48 |
| mattw4 | fungi: I pasted the role activity above. It seems to work just fine then moves on | 21:49 |
| clarkb | ya its almost like devstack isn't in the find list | 21:50 |
| clarkb | which might happen due to perms? | 21:50 |
| fungi | oh, maybe if it's not readable | 21:50 |
| mattw4 | I had devstack in my required-projects list and in tenant config. That should be it, right? The devstack dir i nsrc/opendev is owned by root (as are the other there), and the repos in /opt/stack are all owned by stack. | 21:51 |
| *** pcaruana has quit IRC | 21:51 | |
| *** EvilienM is now known as EmilienM | 22:42 | |
| fungi | baffled by this unit test result: http://logs.openstack.org/70/662870/3/check/tox-py36/74aaa81/job-output.txt.gz#_2019-06-05_17_54_38_388095 | 23:02 |
| fungi | i'm guessing something about the change is causing some sort of infinite looping/recursion in a test? | 23:02 |
| fungi | (this is for the password lookup plugin addition) | 23:03 |
| fungi | the tests i add in 662870 aren't any of the one which ran immediately prior to the exception being raised in subunit | 23:05 |
| fungi | alternatively, has our debug output for unit tests simply outgrown subunit's capacity? | 23:06 |
| fungi | also, i add 5 plugin tests... are those perhaps far too expensive to be so liberal with? | 23:06 |
| fungi | (i notice we're missing a bunch of coverage for the previously-allowed plugins) | 23:07 |
| clarkb | I cant get it to load currently but subunit does have a buffer limit | 23:10 |
| clarkb | (and it isnt huge) | 23:10 |
| *** michael-beaver has quit IRC | 23:24 | |
| *** tjgresha has joined #zuul | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!