| *** jamesmcarthur has joined #zuul | 01:28 | |
| *** rlandy|bbl is now known as rlandy | 01:41 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| tristanC | jhesketh_: so, should i start rebasing the zuul-runner on the multi-ansible stack? | 02:35 |
|---|---|---|
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: enable zuul_return to update Ansible inventory https://review.openstack.org/590092 | 02:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 03:51 |
| *** chkumar246 is now known as chandankumar | 03:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:35 |
| *** bjackman_ has joined #zuul | 04:57 | |
| openstackgerrit | Merged openstack-infra/zuul master: Ensure correct lexical sorting of node requests https://review.openstack.org/643352 | 05:47 |
| openstackgerrit | Merged openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 05:47 |
| openstackgerrit | Merged openstack-infra/zuul master: Add missing doc for 'execute_time' in MQTT schema https://review.openstack.org/643069 | 05:47 |
| *** chandankumar has quit IRC | 05:58 | |
| *** swest has joined #zuul | 06:12 | |
| *** saneax has joined #zuul | 06:14 | |
| *** swest has quit IRC | 06:15 | |
| *** swest has joined #zuul | 06:17 | |
| *** chandankumar has joined #zuul | 06:22 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 06:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Wait for command processor thread to stop https://review.openstack.org/641854 | 07:03 |
| *** saneax has quit IRC | 07:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove some thread join timeouts https://review.openstack.org/643517 | 07:24 |
| tristanC | win 5 | 07:24 |
| tristanC | oops :) | 07:24 |
| tobiash | ? | 07:24 |
| tobiash | I hope that's not your password ;) | 07:25 |
| tobiash | but that would be a weak pw anyway ;_ | 07:25 |
| tristanC | tobiash: that's an irssi command, its just missing a / prefix | 07:25 |
| tobiash | ah | 07:25 |
| *** yolanda has joined #zuul | 07:30 | |
| tobiash | tristanC: where do you use jmespath? | 07:35 |
| *** pcaruana has joined #zuul | 07:37 | |
| tristanC | tobiash: in some job to process zuul.projects with json_query() filter | 07:38 |
| tobiash | so the json_query filter requires this? | 07:38 |
| tobiash | that's an official part of ansible and is not pulled in as a dep? | 07:39 |
| tobiash | https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#json-query-filter | 07:39 |
| tristanC | tobiash: yes, at least in 2.5, and it's similar to openstacksdk, it's pulled in as a dep but it's used in official part of ansible | 07:40 |
| tristanC | json_query seems to be used in openstack projects too, not sure if they are zuul task though: http://codesearch.openstack.org/?q=json_query&i=nope&files=&repos= | 07:41 |
| tristanC | it's not* pulled in as a dep | 07:41 |
| tobiash | ok, verified that it isn't pulled in into any version of ansible | 07:44 |
| tristanC | tobiash: btw, we are not planning to use fully fledge venv per ansible version, we want to use system-site-packages whenever possible | 07:44 |
| tristanC | tobiash: and it seems like ansible-2.[567] works well with a single set of deps | 07:44 |
| tobiash | tristanC: what's your reasoning for requiring system-site-packages? | 07:47 |
| tristanC | tobiash: so that we only install the deps once, and we can manage them globally | 07:47 |
| tobiash | you mean your deployment specific additional deps? | 07:48 |
| tristanC | for example, we only have one python-paramiko in /usr/lib/python3.5 that gets used by zuul, nodepool and now, the different ansible version that are bundled for the executor | 07:50 |
| tristanC | fwiw, here is a first %install draft that seems to work well: https://softwarefactory-project.io/r/#/c/15219/1/zuul-executor-ansible-25.spec@50 | 07:50 |
| *** gtema has joined #zuul | 07:51 | |
| tristanC | and it seems like you could layer two venv with the pyvenv.cfg file so that you could still share ansible deps without relying on system-site-packages | 07:52 |
| tobiash | hrm, I didn't plan to mess that deep with venv config, would you be ok with adding an optional switch to use system-site-packages? | 07:53 |
| tristanC | last time i checked, using pip with system-site-packages can causes issues | 07:56 |
| tristanC | though it would be nice if it was possible to have the zuul-manage-ansible script install ansible common deps in {ansible_root}/ansible/deps and then use that as a parent for the different version bundle | 07:57 |
| tristanC | not sure if it's possible or that simple though | 07:57 |
| tristanC | but that would improve disk/mem usage if we could have only one openstacksdk instead of 3 | 07:58 |
| tristanC | or any other heavy dependencies ansible might need | 07:59 |
| tobiash | tristanC: can we do that as a later improvement to the stack? | 07:59 |
| tobiash | I just tried system-site-packages and both venvs ended up with exactly the same size | 08:00 |
| tobiash | tristanC: in the meantime the zuul-manage-ansible script already can handle site specific extra deps, they just won't be shared | 08:00 |
| tristanC | tobiash: sure, though if it's possible, then that might render the per ansible version requirements line obsolote, at least for the extra deps | 08:01 |
| tristanC | i'm not sure what's the use case to have per ansible version requirements | 08:01 |
| tristanC | if your job need openstacksdk for upload-swift, or openshift for kubectl connection, then you'll have to add those to each version | 08:01 |
| tobiash | because you also might want to blacklist specific patch versions | 08:01 |
| tristanC | that sounds complex, would you do that for the extra reqs? or this is just for the final ansible version? | 08:02 |
| tristanC | e.g. something like openstacksdk<0.19 for ansible-2.5 but openstacksdk>0.23 for ansible-2.7? | 08:03 |
| tristanC | tobiash: (if your system-site-packages doesn't have any, or the wrong version, of the requirements needed then the venv doesn't benefit from that) | 08:03 |
| tobiash | yes, that must be possible | 08:04 |
| tobiash | you can add extra deps by setting the env var ANSIBLE_25_EXTRA_PACKAGES when running zuul-manage-ansible | 08:05 |
| tobiash | but just noticed that doesn't seem to be documented ;) | 08:05 |
| tobiash | but I agree that a common extra packages makes sense | 08:05 |
| tobiash | that should be used unless there are specific version requirements per ansible | 08:05 |
| tobiash | (which could happen e.g. to ara) | 08:06 |
| tobiash | I'll look into that, looks like I have to rebase anyway due to merge conflicts | 08:08 |
| tristanC | tobiash: do you think it will take time to merge multi-ansible? | 08:08 |
| tobiash | depends on reviews ;) | 08:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Manage ansible installations within zuul https://review.openstack.org/631930 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Validate ansible installations on startup https://review.openstack.org/637418 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Make ansible version configurable https://review.openstack.org/637422 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Install ansible during executor startup if needed https://review.openstack.org/640644 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Run tox remote concurrent https://review.openstack.org/640654 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support ansible 2.6 https://review.openstack.org/631931 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support ansible 2.7 https://review.openstack.org/631932 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Switch default ansible version to 2.7 https://review.openstack.org/637424 | 08:10 |
| tobiash | that's a pure rebase to fix merge conflicts in zuul.yaml ^ | 08:10 |
| *** jpena|off is now known as jpena | 08:11 | |
| tristanC | well i've picked the stack in our master zuul version of zuul and so far so good | 08:11 |
| tobiash | I'll add the common dependency definition on top I think | 08:12 |
| tristanC | unfortunately the patch doesn't apply cleanly in the spec file, it fail with "File zuul/ansible/lookup/url.py is not a regular file -- refusing to patch" | 08:13 |
| tobiash | did you patch that file locally? | 08:13 |
| tristanC | thus it would help to get the long rename list first (631930) | 08:13 |
| tristanC | tobiash: it's the version of patch used by rpmbuild that doesn't support rename properly iiuc | 08:15 |
| tobiash | ah | 08:15 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 08:23 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Run tox-py35 jobs on xenial https://review.openstack.org/643530 | 08:23 |
| *** pcaruana has quit IRC | 08:24 | |
| *** pcaruana|afk| has joined #zuul | 08:24 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: wip: Spec for allowing circular dependencies https://review.openstack.org/643309 | 08:30 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Support common ansible dependencies https://review.openstack.org/643534 | 08:52 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend https://review.openstack.org/643536 | 08:59 |
| *** saneax has joined #zuul | 08:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: ansible-config: add common extra dependencies https://review.openstack.org/643537 | 09:01 |
| *** rfolco has quit IRC | 09:07 | |
| *** logan- has quit IRC | 09:23 | |
| *** logan- has joined #zuul | 09:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support common ansible dependencies https://review.openstack.org/643534 | 09:42 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add missing docs for zuul-manage-ansible https://review.openstack.org/643544 | 09:42 |
| tobiash | tristanC: this adds docs and the common dependencies ^ | 09:42 |
| tobiash | tristanC: you might want to rebase 643534 on top of these | 09:42 |
| tobiash | I meant 643537 | 09:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: ansible-config: add common extra dependencies https://review.openstack.org/643537 | 09:45 |
| tobiash | tristanC: just did it ^ | 09:45 |
| tristanC | tobiash: thank you, that looks good to me | 09:58 |
| *** saneax has quit IRC | 10:01 | |
| *** saneax has joined #zuul | 10:01 | |
| *** rfolco has joined #zuul | 10:05 | |
| *** rfolco has quit IRC | 10:09 | |
| *** rfolco has joined #zuul | 10:11 | |
| bjackman_ | I have an issue again where I have a queued job not running, while no other jobs are running | 10:43 |
| bjackman_ | Can see 10 jobs in my gate queue in the web UI - 9 show "success" and one shows "queued" | 10:43 |
| bjackman_ | Have collected logs but could anyont point me in the right direction for searching for what's gone wrong? | 10:44 |
| bjackman_ | Ahh... the job did actualyl run, then got "WARNING zuul.AnsibleJob: [build: 07e857d5686b4aecac9b4bf0b33a55df] Ansible timeout exceeded: 60" | 10:56 |
| bjackman_ | (Or, it started to run) | 10:57 |
| *** saneax has quit IRC | 11:07 | |
| *** saneax has joined #zuul | 11:07 | |
| badboy | can I use <br> or something in the pipeline description? | 11:30 |
| *** saneax has quit IRC | 11:49 | |
| *** saneax has joined #zuul | 11:49 | |
| *** saneax has quit IRC | 11:50 | |
| *** saneax has joined #zuul | 11:51 | |
| *** jpena is now known as jpena|lunch | 11:59 | |
| *** saneax has quit IRC | 12:11 | |
| *** saneax has joined #zuul | 12:12 | |
| *** panda is now known as panda|lunch | 12:15 | |
| *** jamesmcarthur has joined #zuul | 12:20 | |
| *** rlandy has joined #zuul | 12:26 | |
| *** pcaruana has joined #zuul | 12:27 | |
| *** pcaruana|afk| has quit IRC | 12:28 | |
| tobiash | badboy: I guess yes, but no idea if that get interpreted | 12:35 |
| *** jamesmcarthur has quit IRC | 12:36 | |
| *** EmilienM is now known as EvilienM | 12:37 | |
| badboy | tobiash: it does not get interprated | 12:47 |
| *** jamesmcarthur has joined #zuul | 12:48 | |
| dmsimard | Yesterday's 45-patch-stack from Nova ended up paralyzing the OpenStack Zuul for ~30 minutes and the RDO/SF Zuul for even longer (because we have less mergers). Are there any known opportunities to improve this other than to add more mergers ? | 12:50 |
| pabelanger | dmsimard: 6 hours for sf.io, and yah. I think I'm going to work on it today, if we can come up with a solution moving forward | 12:51 |
| dmsimard | pabelanger: have any ideas yet ? | 12:52 |
| pabelanger | in the case of rdo, I don't think we actually needed hit the mergers, because zuul tenant configuration is setup not to load any jobs from that project, or do we have any jobs configured to run on changes from it. | 12:52 |
| pabelanger | It is only in the system so depends-on would work properly | 12:52 |
| pabelanger | dmsimard: I believe tobiash, corvus or clarkb may also have ideas | 12:53 |
| pabelanger | but yes, I would like to see if we can fix this | 12:53 |
| pabelanger | as ansible-network was blocked because of rdoproject tenant | 12:53 |
| dmsimard | pabelanger: this isn't a config problem imo, it's a performance one :p | 12:54 |
| *** gtema has quit IRC | 12:54 | |
| pabelanger | dmsimard: my thoughts are, if the project is setup not to load any configuration from it, I am unsure why it needs to go to merger and ask for configuration changes, if I understand this issue properly | 12:55 |
| pabelanger | we might be able to by-pass the merger altogether | 12:55 |
| pabelanger | dmsimard: in the case of zuul mergers in rdocloud, I can't imagine ceph was very happy when this happened, given all the disk IO mergers use | 12:57 |
| *** gtema has joined #zuul | 12:57 | |
| mordred | tobiash: stack looks great to me - great work! | 12:58 |
| *** bjackman_ has quit IRC | 12:59 | |
| *** saneax has quit IRC | 13:01 | |
| *** bjackman_ has joined #zuul | 13:01 | |
| *** saneax has joined #zuul | 13:02 | |
| dmsimard | Found a bug in zuul-web when two projects on different connections are called the same, created a story: https://storyboard.openstack.org/#!/story/2005244 | 13:03 |
| mordred | pabelanger: yeah- I think bypassing the merge step for projects that we're not loading config from seems like a good optimization | 13:03 |
| mordred | dmsimard: oh nice! | 13:04 |
| *** jpena|lunch is now known as jpena | 13:04 | |
| mordred | dmsimard: and of course, by 'oh nice' I mean "doh" | 13:04 |
| dmsimard | mordred: eh, it's not terrible. projects shouldn't be called the same to begin with but that's history | 13:06 |
| mordred | dmsimard: we're showing connection in the list already, so we at least have some of the info there already | 13:06 |
| mordred | dmsimard: yeah. but we do support a full-name for projects if they do - so might as well support it here too | 13:06 |
| pabelanger | mordred: any idea where I should be looking to apply that patch? | 13:35 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/zuul-jobs master: Fix exec quoting in release remove-gpgkey role https://review.openstack.org/643585 | 13:36 |
| *** pcaruana has quit IRC | 13:36 | |
| pabelanger | looking at rdoproject configuration, it does seem jobs are running against openstack/nova, trying to see if that is correct | 13:37 |
| pabelanger | but now dealing with the 2nd time in 24ths that openstack/nova has rebased a 48 stack of changes. | 13:38 |
| dmsimard | yeah this is really problematic | 13:38 |
| pabelanger | 24ths/24hrs* | 13:38 |
| dmsimard | Is there no way to multi-thread the merging ? | 13:39 |
| dmsimard | The fact that everything is single threaded and sequential probably does not help | 13:39 |
| fungi | dmsimard: you could probably just run multiple mergers on the same server | 13:44 |
| fungi | if your dedicated merger isn't using all the resources available on that system | 13:45 |
| tobiash | mordred: thanks :) | 13:45 |
| dmsimard | fungi: yeah right now I suppose we're doing similar to openstack-infra and we run multiple small mergers | 13:45 |
| dmsimard | but the rate at which these small mergers chew through the backlog is just really slow | 13:46 |
| clarkb | re multithreaded it is intentionally scale out via the gearman bus | 13:48 |
| clarkb | the big constraint is one merger per repo iirc | 13:48 |
| pabelanger | So, looking at zuul configuration, for 3pci, does it make sense to have an upstream project in a merge-check pipeline? I guess that would mean job would run, but fail to properly be merged at some point | 13:48 |
| fungi | clarkb: oh, so one merger gets dedicated to the nova repo even if there are multiple changes to do merging tasks for? | 13:49 |
| fungi | pabelanger: we don't even run a merge-check pipeline any longer | 13:49 |
| clarkb | fungi: sorry no I mean you cant have two threads talking to one real repo on disk | 13:49 |
| pabelanger | fungi: ah, didn't know that | 13:49 |
| clarkb | fungi: all of our mergershave distinct repos on disk | 13:50 |
| fungi | and no, i don't think performing merge checks on a repository your zuul doesn't control is especially useful to anyone | 13:50 |
| fungi | clarkb: oh, got it. right you'd need a separate tree in the fs with distinct copies of the repos | 13:50 |
| clarkb | ya | 13:50 |
| fungi | for each merger daemon | 13:50 |
| tobiash | I think we maybe be doing more merger call than needed. At least the last time I looked at this we were doing those merge calls when having changes A -> B -> C. Just change C Will create these merges: A, A, AB, A, AB, ABC | 13:50 |
| fungi | still probably doable | 13:50 |
| tobiash | at least that was like one year ago, not sure if it's still the case | 13:51 |
| clarkb | tristanC: I think that is still the case | 13:51 |
| tobiash | so I think we should look into how this can be optimized | 13:52 |
| tobiash | to me it looks like it does the merge call also for the non-live items in the independent pipeline | 13:52 |
| clarkb | tobiash: it has to for those | 13:52 |
| tobiash | currently that's needed so we have the dynamic layout of the parent item which is non-live | 13:53 |
| clarkb | since the non live items affect git state in the live item | 13:53 |
| tobiash | but we could extend the merger api to get all layout changes of a list of items in one call | 13:53 |
| pabelanger | fungi: thanks, testing out dropping it now | 13:54 |
| tobiash | we could tell the merger merge a,b,c (which we anyway already do in the last step) and the merger currently only returns layout changes for c | 13:54 |
| tobiash | if the merger could return layout changes for all changes in the dependency chain at once we could reduce that to one merger call | 13:54 |
| clarkb | ya it doesnt have to do the multiple iterations tristanC deacribes | 13:57 |
| tobiash | the number of merges per change is currently (n^2 +n)/2 which doesn't really matter for small dependency chains but can really get a problem with large rebases | 13:57 |
| tobiash | in github that's not that much of a problem because of the different workflow | 13:58 |
| *** jamesmcarthur has quit IRC | 13:59 | |
| clarkb | sorry I keep tab completing tobiash to tristanC | 13:59 |
| tobiash | no worries I know that I was meant ;) | 13:59 |
| fungi | pabelanger: part of the reason we stopped running a merge-check pipeline in zuul is because newer versions of gerrit added its own merge checker anyway so having zuul spontaneously leave comments on any change which started to merge-conflict with one that just merged was redundant | 14:03 |
| fungi | because gerrit ought to show that to you anyway on its own | 14:03 |
| pabelanger | ack | 14:05 |
| tobiash | yay, someone broke a big repo for zuul by creating a branch containing ');' at the end of the name... | 14:16 |
| *** pcaruana has joined #zuul | 14:17 | |
| clarkb | check-ref-format says that isn't valid | 14:19 |
| *** jhesketh_ has quit IRC | 14:20 | |
| clarkb | though reading the rules i don't see why | 14:20 |
| tobiash | git didn't complain | 14:20 |
| tobiash | but zuul failed to process that repo | 14:20 |
| clarkb | `git check-ref-format 'foo);'` exits zero for me | 14:21 |
| clarkb | maybe I need the full refname | 14:21 |
| clarkb | ah yup that is it need to make it a path then it exits 0 | 14:22 |
| *** bjackman_ has quit IRC | 14:22 | |
| tobiash | actually this is valid: git check-ref-format 'refs/remotes/foo);' | 14:23 |
| clarkb | yup I reread the rules and the / is required somewhere | 14:23 |
| clarkb | so even foo/bar); is valid | 14:23 |
| clarkb | so a gitpython or zuul escaping bug | 14:24 |
| dmsimard | tobiash: I think you've summarized the issue well with your A -> B -> C example | 14:30 |
| dmsimard | ++ | 14:30 |
| *** jamesmcarthur has joined #zuul | 14:31 | |
| tobiash | dmsimard: yeah and I think it is solvable by some minor or medium architectural change | 14:31 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix exec quoting in release remove-gpgkey role https://review.openstack.org/643585 | 14:36 |
| tobiash | clarkb: oh it wasn't ); that broke zuul, it was an '@' sign | 14:41 |
| tobiash | (yes, someone used his mail address as branch name) | 14:41 |
| pabelanger | Yah, I haven't seen a way to do ACLs on branch creation in github | 14:44 |
| pabelanger | would be nice | 14:45 |
| *** jhesketh_ has joined #zuul | 14:46 | |
| tobiash | I think we might restrict that by using a pre-receive hook | 14:47 |
| tobiash | we also had problems with windows builds because someone pusht the branches foo and Foo at the same time | 14:48 |
| *** nhicher has joined #zuul | 14:48 | |
| *** saneax has quit IRC | 14:49 | |
| *** jhesketh has quit IRC | 14:49 | |
| *** jhesketh has joined #zuul | 14:49 | |
| *** saneax has joined #zuul | 14:49 | |
| *** jhesketh_ has quit IRC | 14:51 | |
| *** hashar has joined #zuul | 14:52 | |
| *** jhesketh has quit IRC | 14:54 | |
| corvus | clarkb, mordred, tobiash, fungi: can you please review https://review.openstack.org/643167 soonish? when that lands, i can send an email to the list and declare the whole project done :) | 14:56 |
| tobiash | corvus: yes, will do | 14:56 |
| clarkb | http://logs.openstack.org/67/643167/1/check/tox-docs/135bded/html/docker-image.html rendered version link if anyone prefers to read that | 14:57 |
| mordred | corvus: I'm on a phone call - my last skim looked great, but feel free to land without me | 14:59 |
| *** jhesketh has joined #zuul | 15:03 | |
| corvus | tobiash, clarkb, pabelanger, dmsimard: https://review.openstack.org/163922 is a small optimization, which, if we could fix, would speed up nova merge times some. i agree that tobiash's description of how we we could be a bit more holistic in using merge data may be a potential big improvement, though i don't think that change will be simple. | 15:04 |
| corvus | i don't have time to take another try at 163922 now, so if anyone wants to take that over, feel free. but i haven't dropped it because i think it's still relevant. | 15:04 |
| pabelanger | k, let me get some coffee and look at it | 15:05 |
| dmsimard | wow, 2015 ? :D | 15:05 |
| dmsimard | a prehistoric commit haha | 15:05 |
| corvus | dmsimard: it's still 6 digits :) though it is my oldest open change | 15:05 |
| clarkb | corvus: noted a few things on the docs change, Only one is really necessary to chagne i think | 15:06 |
| corvus | wow, tobiash recently pointed out that i can abandon my second oldest change! \o/ | 15:07 |
| tobiash | :) | 15:07 |
| corvus | clarkb: replied | 15:10 |
| Shrews | corvus: that's pretty neat! i only found a few spelling errors messing with my OCD | 15:13 |
| Shrews | corvus: re: 167, that is | 15:14 |
| openstackgerrit | Merged openstack-infra/zuul master: web: support more than one job in the filter https://review.openstack.org/643385 | 15:15 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to mirror a git repository to a remote git server https://review.openstack.org/643394 | 15:36 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Do not merge: test zuul-jobs-upload-git-mirror job https://review.openstack.org/643437 | 15:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Document how to build a buildset registry https://review.openstack.org/643167 | 15:37 |
| corvus | all nits addressed | 15:37 |
| *** saneax has quit IRC | 15:46 | |
| *** saneax has joined #zuul | 15:47 | |
| tobiash | clarkb: the @ bug is not a bug in zuul, but a bug in GitPython. This is an attempt to fix that: https://github.com/gitpython-developers/GitPython/pull/854 | 15:58 |
| *** [GNU] has quit IRC | 15:58 | |
| tobiash | corvus, mordred, clarkb: what do you think about switching (maybe partly) to pygit2? This works on top of libgit2 and doesn't spawn a process for every operation. It would probably also return better exceptions so we would not rely on parsing the commandline for detecting a few error cases. | 16:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Don't check out a commit if it is already checked out https://review.openstack.org/163922 | 16:04 |
| clarkb | do they publish anylinux wheels? | 16:04 |
| clarkb | if not we may need to double check libgit2 is availalbe on common distros | 16:04 |
| pabelanger | corvus: ^updated your patch to deal with gitpython exception I was getting locally, but unsure how to write a proper test | 16:05 |
| pabelanger | seem if repo has detached head, you cannot access the repo.head.reference | 16:06 |
| corvus | tobiash: there's 2 things we need from a git library, and at the time, only gitpython provided them -- direct access to git objects (so that once a merge was performed, accessing refs and files is fast), and being able to perform 'git merge' in exactly the way cgit does with all the same options. if pygit2 does that now, i'm open to it (modulo clarkb's concerns). | 16:06 |
| mordred | clarkb: it's available in stretch at least, so would work in the python:slim docker images - other than that I don't know | 16:06 |
| corvus | tobiash: wait, you said "process for every operation". you know that doesn't happen, right? | 16:07 |
| clarkb | I'm checking pypi | 16:07 |
| tobiash | well, process for most operations ;) | 16:07 |
| corvus | tobiash: there are only a few operations we do that spawn a process, and they're pretty slow anyway. | 16:07 |
| clarkb | There are only windows wheels so ya we'll need to be able to install dev headers and the lib system wide | 16:07 |
| tobiash | a few weeks ago I experimented with it for an hour or so and using it at some places improved the test case runtime quite a bit | 16:08 |
| tobiash | but I know that in production that won't make much of a difference | 16:09 |
| clarkb | looks like libgit2 is widely available so no issue there (I checked ubuntu suse fedora) | 16:09 |
| tobiash | also debian has it | 16:10 |
| tobiash | yah, it's probably not worth switching completely | 16:11 |
| tobiash | I think the most important thing I saw during experimenting was improved test case performance | 16:11 |
| tobiash | so I set it aside at that time | 16:12 |
| tobiash | it just came to my mind because of that '@' char bug that tells me that GitPython re-implements quite some git stuff in python | 16:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Allow group read permissions on project keys directory https://review.openstack.org/642913 | 16:18 |
| *** kmalloc is now known as needscoffee | 16:25 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Document how to build a buildset registry https://review.openstack.org/643167 | 16:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Don't check out a commit if it is already checked out https://review.openstack.org/163922 | 16:37 |
| *** hashar has quit IRC | 16:49 | |
| *** rlandy is now known as rlandy|biab | 17:00 | |
| *** jamesmcarthur has quit IRC | 17:00 | |
| *** jamesmcarthur has joined #zuul | 17:01 | |
| tobiash | corvus: looks like I was tio slow for 167 but lgtm :) | 17:04 |
| *** jpena is now known as jpena|off | 17:21 | |
| *** rfolco is now known as rfolco|doctor | 17:31 | |
| *** rlandy|biab is now known as rlandy | 17:51 | |
| *** gtema has quit IRC | 18:07 | |
| corvus | tobiash: did you want to leave a vote on https://review.openstack.org/643537 ? | 18:12 |
| *** saneax has quit IRC | 18:13 | |
| corvus | the entire multi-ansible stack is ready to land; i omitted a +W on the first change though. do we want to land it today, or first thing next week? | 18:14 |
| corvus | (i will be leaving the keyboard in an hour and won't be back until monday) | 18:14 |
| clarkb | probably better to merge next week then so if people end up running it there are peolle around to help with any issues? | 18:17 |
| *** jamesmcarthur has quit IRC | 18:20 | |
| tobiash | corvus: ++ | 18:32 |
| tobiash | \o/ | 18:33 |
| fungi | "...using the full power of Zuul’s speculative execution." i love the phrasing! | 18:35 |
| tobiash | corvus: I placed a -w on the first one so this won't get accidentally merged before monday | 18:36 |
| clarkb | one thing I dont seeafter looking is virtualenv in the requirements file? I may just be missing it | 18:44 |
| tobiash | clarkb: I think I've put it in bindep | 18:45 |
| fungi | i'm probably lacking some context, but could it use the built-in venv module in python 3? | 18:45 |
| tobiash | fungi: no, that didn't work as it didn't install a new enough pip | 18:45 |
| clarkb | fungi there is a comment in the code explaining that is broken if used from within a venv | 18:45 |
| fungi | ooh, can you point to that? i'm interested in the details | 18:46 |
| clarkb | tobiash: hrm I guess bindep also works though it is a python dep and could go in requirements | 18:46 |
| tobiash | clarkb: I've put it into bindep because we call it as a process | 18:46 |
| tobiash | it is no python dep | 18:46 |
| fungi | for some of my python3-only projects i've moved to just using venv, including for tox (the via tox-venv plugin) | 18:46 |
| tobiash | fungi: venv was the first thing I tried but it was impossible to use it in the openstack test environment because of too old pip that doesn't work with two mirror urls | 18:47 |
| clarkb | tobiash: well it is still a python dep like ansible or zuul even though we also run those as commands | 18:47 |
| tobiash | that inhibited upgrading pip | 18:47 |
| tobiash | clarkb: that depends, the virtualenv binary in older ubuntu actually comes from py27 | 18:48 |
| fungi | interesting, i thought most of our images included the latest pip version | 18:48 |
| tobiash | unfortunately not because venv has been my favorite actually | 18:49 |
| clarkb | I thought so too fwiw | 18:50 |
| clarkb | (easy enough to check via the pip logs of a job) | 18:50 |
| tobiash | clarkb: it's buried in some of the oder revisions of the first change ;) | 18:50 |
| tobiash | it took me quite a few revisions to get it working in the openstack ci | 18:50 |
| tobiash | ;) | 18:50 |
| clarkb | tobiash: re python2 virtualenv I don't think thatmatters? I guess if you run virtualenv without -p it defaults to the python it was instaleld under so that could change behavior for people | 18:52 |
| tobiash | clarkb: correct we explicitly run with -p so that doesn't matter, we just use the one that we can get from the os | 18:52 |
| clarkb | pip 19.0.3 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7) is what we install on our test images | 18:53 |
| clarkb | that is latest pip | 18:53 |
| clarkb | tobiash: the problem may actually be your use of virtualenv from the system | 18:54 |
| clarkb | tobiash: since virtualenv depends on pip it will potentially use the system pip from system virtlanev? (I dunno the way those two interact has changed several times over the years) | 18:55 |
| clarkb | used to be that virtualenv vendored pip | 18:55 |
| clarkb | then virtualenv was updated to always upgrade pip and setuptools | 18:55 |
| clarkb | and who knows what distro packaging does | 18:55 |
| tobiash | clarkb: the problem was that I tried venv (and requiring it in requirements.txt) first. That didn't work on the xenial nodes because it installed the system pip into the venv with no possible upgrade path because of mirror urls. | 18:55 |
| tobiash | clarkb: using the virtualenv binary then just worked because it installed a recent version of pip | 18:56 |
| clarkb | tobiash: why would virtualenv use the system pip? but even if it did our system pip is latest | 18:56 |
| fungi | yeah, virtualenv vendors a copy of pip | 18:57 |
| clarkb | (I'd like to figure this out because we don't intend to run old pip) | 18:57 |
| fungi | if we're running distro-provided python3 packages, the integrated venv and pip are separated out into additional packages | 18:57 |
| tobiash | clarkb: we could try to put a refactoring change on top of the stack that switches that back to venv if you want to analyze that in more detail | 18:58 |
| clarkb | tobiash: ya I'll push one up | 18:58 |
| corvus | tobiash: ok, assuming you, clarkb, and fungi don't decide to redo the patch series, then maybe you can +W it when you start work monday | 18:58 |
| fungi | i too would be interested to see us not rely on virtualenv for such things, as it's increasingly abandoned and the recent maintainer who has stepped up seems focused mostly on ensuring it works on mac os | 18:59 |
| fungi | but as a transitional thing it seems fine | 18:59 |
| tobiash | corvus: k, will do | 18:59 |
| clarkb | ya I don't think we need to rewrite the stack | 18:59 |
| clarkb | can just do followups to refine it | 18:59 |
| tobiash | yes, I think the virtualenv vs venv won't block us | 19:00 |
| corvus | and if we decide to do venv, i think it's fine to do that as a followup -- but maybe let's hold off on landing anything until it's ready | 19:00 |
| corvus | or at least until we know the plan | 19:00 |
| tobiash | clarkb: btw, diskimage-builder doesn't work in a venv-generated venv | 19:00 |
| corvus | mostly i don't want to land the stack and then immediate have significant changes, since we'll be asking people to change their deployment tooling | 19:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 19:00 |
| fungi | for my part, it's more of a curiosity. since we're python3-only we shouldn't need to rely on virtualenv (and this is part of the reason why upstream maintenance of virtualenv itself has waned heavily in the last year or so) | 19:00 |
| clarkb | my bigger concern here is that the test nodes are not acting the way they are supposed to if they work they way tobiash describes | 19:01 |
| clarkb | and checking things manually just now they don't work they way tobiash describes (they have latest pip) | 19:01 |
| clarkb | so we should sort that out from an image perspective | 19:01 |
| fungi | the original virtualenv authors seem to consider getting the venv module working in python>=3.4 as "job done" and once python2.7 is eol i think the days of the virtualenv module are numbered | 19:02 |
| clarkb | you know I wonder if the issue is that distros don't include venv in their standard python installs | 19:03 |
| fungi | certainly interest in making it work for python 3.x is limited, and interest in keeping it working on gnu/linux platforms as well | 19:03 |
| clarkb | and its getting confused in the system packaging that way? | 19:03 |
| fungi | debian and its derivatives like ubuntu still build it but stick it in a separate python.*-venv package | 19:04 |
| tobiash | clarkb: this was the latest revision that used venv: https://review.openstack.org/#/c/631930/9/zuul/lib/ansible.py | 19:04 |
| fungi | si dunno about other distro families | 19:04 |
| fungi | however, part of the problem i think is that improvements to the venv module get backported to new point releases of earlier python versions but don't get incorporated into the packages of lts distros | 19:05 |
| clarkb | ya so I think what might be happening there is xenial has old venv (not virtualenv) and that doesn't use newer pip and it may be hardcoded to use system package installed pip not pip installed pip which is latest | 19:06 |
| tobiash | oh, I think actually the py36 job was broken, not the py35, so bionic was the problem somehow | 19:07 |
| fungi | so building python myself i run (for 3.5) a 3.5.6 (soon 3.5.7) version but ubuntu xenial keeps selectively backporting critical fixes to 3.5.2 | 19:07 |
| clarkb | it uses https://docs.python.org/3.5/library/ensurepip.html#module-ensurepip which says it uses the bundled pip? | 19:08 |
| fungi | thus the venv and pip functionality integrated via their system python packages is contemporary to when they released in 2016 | 19:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test venv instead of virtualenv https://review.openstack.org/643658 | 19:09 |
| fungi | the dichotomy of distro-packaged python or building python yourself from upstream source | 19:09 |
| tobiash | clarkb: that switches to venv, let's see ^ | 19:09 |
| clarkb | tobiash: you might have to update bindep on that one since venv isn't included in the debuntu distros python installs by default | 19:10 |
| tobiash | could be | 19:10 |
| clarkb | and there isn't a venv package on pypi to undo the distro sillyness | 19:11 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test venv instead of virtualenv https://review.openstack.org/643658 | 19:11 |
| clarkb | thats annoying that they don't include all the stdlib bits by default | 19:11 |
| tobiash | clarkb: and the problem using the venv module directly in python was that it didn't install any pip even if I asked it to | 19:14 |
| clarkb | tobiash: ya that part doesn't surprise me given the way the distros tend to try and control that stuff | 19:15 |
| clarkb | (so now I understand better) | 19:15 |
| fungi | right, the python3-venv package i think expects you'll install python3-pip as well | 19:16 |
| fungi | again, problems which go away if you decide not to bother supporting distro-packages python (unfortunately?) | 19:17 |
| *** gouthamr has quit IRC | 19:17 | |
| *** dmellado has quit IRC | 19:17 | |
| fungi | i sort of respect that rd hat punted over the years and told people not to use their python packages to run their custom applications, but instead build their own python from source and put it somewhere else on the system | 19:18 |
| fungi | teh argument was that the system-incorporated python interpreter was an implementation detail of a number of other system components | 19:19 |
| fungi | and not intended as a general-purpose python interpreter | 19:19 |
| fungi | debian (and therefore ubuntu) get to deal with slightly fanatical opinions on package management, that insist that any package manager which is not the system default package manager should not be provided by default, so as to be able to install a packaged python interpreter but be assured there is no integrated package manager which might work around dpkg/apt | 19:21 |
| fungi | i doubt my opinion is widely shared (outside the debian community anyway), but i feel like if you're going to buy into the distro's packaged python interpreter then you should also stick to the distro's packaged python modules. if your intent is to use pip then you might as well also bring your own python interpreter | 19:22 |
| clarkb | given this situation I think not using venv is probably the most straightforward option for us trying to support different platforms that may have different broken python stdlibs | 19:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Fix parent if no virtualenv exists in system https://review.openstack.org/643665 | 19:27 |
| tobiash | clarkb: this is needed on top of your change to support a system with not installed virtualenv | 19:27 |
| tobiash | clarkb: and I think I agree that installing it via requirements.txt ist the better way compared with bindep | 19:28 |
| tobiash | clarkb: so I think when they are green it makes sense to squash 665 into your change and land it together with the whole stack | 19:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 19:30 |
| tobiash | clarkb: this is using venv instead of virtualenv: http://logs.openstack.org/58/643658/2/check/tox-py35/49aaff1/job-output.txt.gz | 19:30 |
| tobiash | http://logs.openstack.org/58/643658/2/check/tox-py35/49aaff1/job-output.txt.gz#_2019-03-15_19_27_02_324093 | 19:31 |
| *** jamesmcarthur has joined #zuul | 19:31 | |
| tobiash | Shrews: want to land https://review.openstack.org/643530 ? | 19:32 |
| mordred | fungi: fwiw - I agree with you - although it has led me to be very supportive of bringing my own python interpreter rather than to being more aggressive using distro python packages | 19:32 |
| * mordred loves pyenv | 19:32 | |
| clarkb | tobiash: ok | 19:33 |
| fungi | mordred: agreed, i don't use pyenv but i do just build python from source (in part because i want to be able to test against latest point releases of 3.4, 3.5, 3.6, 3.7 and future 3.8) | 19:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 19:34 |
| Shrews | tobiash: umm, something seems off about that. why is the tox-py35 job using a nodeset that doesn't include py35??? | 19:38 |
| Shrews | i feel like changing that to bionic was wrong | 19:38 |
| tobiash | Shrews: the default nodeset changed to bionic | 19:38 |
| mordred | yeah. and the tox-py35 job itself doesn't have any info about nodesets because it's in zuul-jobs | 19:39 |
| fungi | corvus: i left a couple of diagram rendering related notes in 643108 in case you happen to know easy fixes for a follow-up change | 19:39 |
| Shrews | yeah, that's what i'm saying. i don't see a good reason for that | 19:39 |
| mordred | Shrews: for what? | 19:39 |
| mordred | (sorry, just making sure I'm following the concern) | 19:40 |
| Shrews | mordred: shouldn't we fix the base tox-py35 job? | 19:40 |
| *** openstack has joined #zuul | 19:44 | |
| *** ChanServ sets mode: +o openstack | 19:44 | |
| tobiash | we have, but my users are yelling at me about every second that we're spending in the base jobs... | 19:45 |
| tobiash | ;) | 19:45 |
| fungi | i mean, for debian/ubuntu nodes it could just use the deadsnakes repository to pull in prebuilt packages for the desired minor version | 19:45 |
| clarkb | and hope the networking between $cloud and launchpad ppas works | 19:46 |
| clarkb | and that the ppa keyserverisn't dead like it was yesterday | 19:46 |
| * clarkb grumbles about keyservers | 19:46 | |
| fungi | clarkb: caching proxies and mirrors! the solution to EVERY problem | 19:46 |
| fungi | oh, and not rtetrieving keys from pgp keyservers in ci jobs, that's just sad | 19:47 |
| clarkb | ya baking in public keys to where we want ot trust them is much better | 19:47 |
| fungi | i've been reminding people of that for years. if you're going to embed a keyid, just embed the full key | 19:48 |
| fungi | you have to change either when the key changes anyway | 19:49 |
| clarkb | ++ | 19:49 |
| fungi | i mean, even when publishing details on signing keys, publish the full key alongside the keyid or fingerprint | 19:49 |
| fungi | it's maybe one or two extra lines of code to include it | 19:50 |
| fungi | keyservers are great for bootstrapping the web of trust and sharing signatures on keys, but they're dicey as a key storage medium | 19:51 |
| tobiash | maybe someone wants to review https://review.openstack.org/632550 ? It fixes undefined behavior and makes it an error to redefine a pipeline under the same name. | 19:52 |
| mordred | tobiash: done! | 19:53 |
| *** gouthamr has joined #zuul | 19:53 | |
| tobiash | mordred: that was quick! | 19:53 |
| tobiash | thanks :) | 19:53 |
| tobiash | and 640609 is a refactoring to handle job canceling the same way in a central place | 19:55 |
| *** pcaruana has quit IRC | 19:59 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 19:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 20:00 |
| openstackgerrit | Merged openstack-infra/nodepool master: Run tox-py35 jobs on xenial https://review.openstack.org/643530 | 20:01 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Use base_prefix instead of exec_prefix to find virtualenv https://review.openstack.org/643672 | 20:04 |
| *** jamesmcarthur has quit IRC | 20:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Use base_prefix instead of exec_prefix to find virtualenv https://review.openstack.org/643672 | 20:17 |
| tobiash | clarkb: that should also work now for the zuul-stream-functional tests ^ | 20:17 |
| tobiash | they installed zuul to /usr/local so they couln't find the virtualenv executable | 20:18 |
| clarkb | cool | 20:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Raise an error if pipeline is defined twice https://review.openstack.org/632550 | 20:21 |
| *** jamesmcarthur has joined #zuul | 20:24 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 20:25 |
| *** dmellado has joined #zuul | 20:29 | |
| *** gouthamr_ has joined #zuul | 20:29 | |
| tobiash | clarkb: I think that should do it ^ | 20:36 |
| mordred | +A | 20:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test simpler way to find virtualenv https://review.openstack.org/643679 | 20:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test simpler way to find virtualenv https://review.openstack.org/643679 | 20:55 |
| tobiash | mordred, clarkb: that might be even better and could be squashed into parent if it's green ^ | 20:55 |
| tobiash | that should guarantee that we find and use the virtualenv we installed with the requirements | 20:56 |
| mordred | tobiash: ooh. yeah. I think that works | 20:57 |
| *** jamesmcarthur has quit IRC | 20:59 | |
| *** rlandy has quit IRC | 21:12 | |
| *** rfolco|doctor has quit IRC | 21:15 | |
| *** hashar has joined #zuul | 21:17 | |
| *** gouthamr has quit IRC | 21:22 | |
| *** jamesmcarthur has joined #zuul | 21:22 | |
| *** jamesmcarthur has quit IRC | 21:27 | |
| pabelanger | tobiash: is ansible-config.conf to be user facing? Or something only zuul knows about? | 21:54 |
| tobiash | pabelanger: only zuul knows about | 22:15 |
| *** jamesmcarthur has joined #zuul | 22:24 | |
| *** hashar has quit IRC | 22:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!