| *** jamesmcarthur has quit IRC | 00:01 | |
| *** jamesmcarthur has joined #openstack-infra | 00:07 | |
| *** wolverineav has quit IRC | 00:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: add job's tags to the rpc job_list method https://review.openstack.org/633653 | 00:08 |
|---|---|---|
| *** wolverineav has joined #openstack-infra | 00:08 | |
| *** lseki has quit IRC | 00:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org https://review.openstack.org/641155 | 00:11 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 00:11 |
| *** wolverineav has quit IRC | 00:13 | |
| *** markvoelker has joined #openstack-infra | 00:17 | |
| *** hamzy has joined #openstack-infra | 00:19 | |
| *** markvoelker has quit IRC | 00:21 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 00:25 |
| gmann | is this fixed 'ERROR! 'chdir' is not a valid attribute for a Task' | 00:26 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 00:27 |
| gmann | found it, 644696. need to recheck tempest fix | 00:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner https://review.openstack.org/607079 | 00:31 |
| *** diablo_rojo has quit IRC | 00:48 | |
| *** markvoelker has joined #openstack-infra | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:50 | |
| *** slaweq has joined #openstack-infra | 00:56 | |
| *** gyee has quit IRC | 00:57 | |
| *** slaweq has quit IRC | 01:00 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add job and integration tests for the upload-git-mirror role https://review.openstack.org/643435 | 01:14 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to mirror a git repository to a remote git server https://review.openstack.org/643394 | 01:14 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Do not merge: test zuul-jobs-upload-git-mirror job https://review.openstack.org/643437 | 01:15 |
| *** mriedem_away has quit IRC | 01:15 | |
| *** jamesmcarthur has joined #openstack-infra | 01:16 | |
| *** jamesmcarthur has quit IRC | 01:18 | |
| *** jamesmcarthur has joined #openstack-infra | 01:19 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| dmsimard | ianw, corvus: for ^ I was wondering if Zuul really had the entire git repo checked out in ~/src/? I suppose we need the full repository, not a shallow copy | 01:26 |
| dmsimard | with all branches and refs ? | 01:26 |
| dmsimard | and tags* | 01:26 |
| clarkb | yes it isnt shallow | 01:27 |
| clarkb | shallow clones break in all sorts of fun ways. easier to give people the full repo | 01:27 |
| clarkb | the repo caches on images help a lot | 01:28 |
| dmsimard | ok, I set up the role to got a git push --mirror from the src repository on disk but now I'm thinking we might want to do a fresh clone from the repo and then mirror that | 01:29 |
| *** ricolin has joined #openstack-infra | 01:29 | |
| clarkb | whats the difference? | 01:29 |
| *** whoami-rajat has joined #openstack-infra | 01:29 | |
| dmsimard | the difference is that the git repo becomes an argument :D | 01:31 |
| dmsimard | clone this, mirror it there | 01:31 |
| dmsimard | instead of "mirror the repository you are testing right now to that location" | 01:31 |
| clarkb | ya I guess the secpnd is finee if you trust zuul | 01:31 |
| clarkb | but more generically useful if you font assume that | 01:32 |
| *** jamesmcarthur has joined #openstack-infra | 01:33 | |
| corvus | dmsimard: the way you wrote it is fine; please don't do a new clone. | 01:38 |
| corvus | this is intended to run in post after changes merge to a repo, so the source repo doesn't need to be an argument. the dest repo can be one of course. | 01:38 |
| dmsimard | ack, thanks | 01:42 |
| *** larainema has joined #openstack-infra | 01:46 | |
| corvus | dmsimard: i left a couple notes on the change | 01:49 |
| *** markvoelker has quit IRC | 01:49 | |
| *** yamamoto has joined #openstack-infra | 02:02 | |
| dmsimard | ack, will address them tomorrow :D | 02:17 |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 02:28 |
| *** trown|outtypewww has quit IRC | 02:31 | |
| *** trown has joined #openstack-infra | 02:33 | |
| *** yamamoto has quit IRC | 02:39 | |
| *** yamamoto has joined #openstack-infra | 02:44 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** psachin has joined #openstack-infra | 02:57 | |
| *** hwoarang has quit IRC | 03:03 | |
| *** hwoarang has joined #openstack-infra | 03:04 | |
| *** yamamoto has joined #openstack-infra | 03:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 03:13 |
| *** apetrich has quit IRC | 03:14 | |
| *** ykarel|away has joined #openstack-infra | 03:15 | |
| *** ykarel|away is now known as ykarel | 03:23 | |
| *** sthussey has quit IRC | 03:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 03:36 |
| *** dave-mccowan has quit IRC | 03:39 | |
| *** ramishra has joined #openstack-infra | 03:41 | |
| *** hwoarang has quit IRC | 03:44 | |
| *** hwoarang has joined #openstack-infra | 03:45 | |
| *** wolverineav has joined #openstack-infra | 03:47 | |
| *** tkajinam has quit IRC | 03:54 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** tkajinam has joined #openstack-infra | 03:57 | |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:01 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** pfallenop has quit IRC | 04:07 | |
| *** wolverineav has quit IRC | 04:11 | |
| *** jamesmcarthur has joined #openstack-infra | 04:27 | |
| *** psachin has quit IRC | 04:34 | |
| *** psachin has joined #openstack-infra | 04:41 | |
| *** janki has joined #openstack-infra | 04:47 | |
| *** larainema has quit IRC | 05:12 | |
| *** wolverineav has joined #openstack-infra | 05:15 | |
| *** wolverineav has quit IRC | 05:18 | |
| *** markvoelker has joined #openstack-infra | 05:22 | |
| *** markvoelker has quit IRC | 05:26 | |
| *** ykarel_ has joined #openstack-infra | 05:43 | |
| *** raukadah is now known as chandankumar | 05:45 | |
| *** ykarel has quit IRC | 05:45 | |
| *** janki has quit IRC | 05:50 | |
| *** jbadiapa has joined #openstack-infra | 05:51 | |
| *** markvoelker has joined #openstack-infra | 05:53 | |
| *** janki has joined #openstack-infra | 05:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: implement prep-workspace https://review.openstack.org/607082 | 05:57 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add configuration schema https://review.openstack.org/640672 | 05:57 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add execute sub-command https://review.openstack.org/630944 | 05:57 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: add command line interface https://review.openstack.org/644770 | 05:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 06:01 |
| *** ykarel_ is now known as ykarel | 06:07 | |
| *** wolverineav has joined #openstack-infra | 06:09 | |
| *** wolverineav has quit IRC | 06:13 | |
| *** jbadiapa has quit IRC | 06:17 | |
| *** sajauddin has joined #openstack-infra | 06:23 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Ensure valid Ansible variable names in config https://review.openstack.org/644574 | 06:28 |
| *** jaosorior has joined #openstack-infra | 06:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 07:06 |
| *** pgaxatte has joined #openstack-infra | 07:11 | |
| *** e0ne has joined #openstack-infra | 07:17 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: implement prep-workspace https://review.openstack.org/607082 | 07:17 |
| *** pcaruana has joined #openstack-infra | 07:20 | |
| *** e0ne has quit IRC | 07:27 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add configuration schema https://review.openstack.org/640672 | 07:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add execute sub-command https://review.openstack.org/630944 | 07:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: add command line interface https://review.openstack.org/644770 | 07:28 |
| *** tosky has joined #openstack-infra | 07:29 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 07:35 | |
| *** roman_g has joined #openstack-infra | 07:40 | |
| *** apetrich has joined #openstack-infra | 07:40 | |
| *** e0ne has joined #openstack-infra | 07:42 | |
| *** jpich has joined #openstack-infra | 07:48 | |
| *** xek_ has joined #openstack-infra | 07:49 | |
| *** dpawlik_ is now known as dpawlik | 07:51 | |
| *** slaweq has joined #openstack-infra | 07:56 | |
| *** ykarel is now known as ykarel|afk | 07:59 | |
| *** ginopc has joined #openstack-infra | 08:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 08:02 |
| *** ccamacho has joined #openstack-infra | 08:11 | |
| *** e0ne has quit IRC | 08:15 | |
| *** ykarel|afk is now known as ykarel | 08:19 | |
| *** helenafm has joined #openstack-infra | 08:20 | |
| *** rpittau|afk is now known as rpittau | 08:27 | |
| *** tkajinam has quit IRC | 08:35 | |
| *** rcernin has quit IRC | 08:38 | |
| *** Lucas_Gray has joined #openstack-infra | 08:40 | |
| *** snapiri has quit IRC | 08:43 | |
| *** jtomasek has quit IRC | 08:49 | |
| *** jpena|off is now known as jpena | 08:53 | |
| rpittau | good morning infra! We're seeing a lot of these errors in the ironic jobs since tonight: http://paste.openstack.org/show/748101/ | 08:58 |
| cmurphy | rpittau: that should be fixed in https://review.openstack.org/644699 , you can recheck | 08:59 |
| rpittau | cmurphy, thank you! :) | 08:59 |
| *** snapiri has joined #openstack-infra | 09:05 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add job parameters listing https://review.openstack.org/644795 | 09:09 |
| *** jtomasek has joined #openstack-infra | 09:10 | |
| *** jbadiapa has joined #openstack-infra | 09:17 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.openstack.org/643731 | 09:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Skip ansible validation for non-ansible tests https://review.openstack.org/644655 | 09:20 |
| *** dtantsur|afk is now known as dtantsur | 09:21 | |
| *** derekh has joined #openstack-infra | 09:25 | |
| *** jaosorior has quit IRC | 09:36 | |
| *** ykarel is now known as ykarel|lunch | 09:43 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Align template formating for reporters https://review.openstack.org/643306 | 09:43 |
| *** ykarel|lunch has quit IRC | 09:48 | |
| *** psachin has quit IRC | 09:48 | |
| *** yamamoto has quit IRC | 09:50 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/irc-meetings master: Add myself as rpm-packaging chair https://review.openstack.org/644806 | 09:50 |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.openstack.org/643731 | 09:54 |
| *** jbadiapa has quit IRC | 09:56 | |
| *** eumel8 has joined #openstack-infra | 09:57 | |
| *** mino_ has joined #openstack-infra | 10:06 | |
| *** mino_ has quit IRC | 10:06 | |
| *** ykarel has joined #openstack-infra | 10:08 | |
| *** ykarel is now known as ykarel|lunch | 10:09 | |
| *** Lucas_Gray has quit IRC | 10:13 | |
| *** Lucas_Gray has joined #openstack-infra | 10:13 | |
| *** jbadiapa has joined #openstack-infra | 10:16 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add myself as rpm-packaging chair https://review.openstack.org/644806 | 10:18 |
| *** electrofelix has joined #openstack-infra | 10:21 | |
| *** psachin has joined #openstack-infra | 10:24 | |
| *** yamamoto has joined #openstack-infra | 10:28 | |
| *** ykarel|lunch is now known as ykarel | 10:33 | |
| *** jaosorior has joined #openstack-infra | 10:35 | |
| *** wolverineav has joined #openstack-infra | 10:41 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 10:45 | |
| *** e0ne has joined #openstack-infra | 10:45 | |
| *** pgaxatte has quit IRC | 10:45 | |
| *** wolverineav has quit IRC | 10:46 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** yamamoto has joined #openstack-infra | 10:47 | |
| *** pgaxatte has joined #openstack-infra | 10:50 | |
| *** dave-mccowan has joined #openstack-infra | 10:54 | |
| *** ykarel is now known as ykarel|afk | 10:55 | |
| *** yamamoto has quit IRC | 10:58 | |
| *** iurygregory is now known as iurygregory|lunc | 10:58 | |
| *** iurygregory|lunc is now known as iury|lunch | 10:58 | |
| *** yolanda has quit IRC | 10:59 | |
| *** yolanda has joined #openstack-infra | 10:59 | |
| *** yamamoto has joined #openstack-infra | 11:00 | |
| *** cdent has joined #openstack-infra | 11:05 | |
| cdent | pabelanger: what's different in ansible 2.7 that requried this https://review.openstack.org/#/c/644699/ ? | 11:11 |
| *** sajauddin has quit IRC | 11:13 | |
| *** jtomasek has quit IRC | 11:14 | |
| *** nicolasbock has joined #openstack-infra | 11:15 | |
| *** janki has quit IRC | 11:16 | |
| *** janki has joined #openstack-infra | 11:16 | |
| *** ykarel|afk is now known as ykarel | 11:17 | |
| aspiers | Would it make sense to configure mod_cache on parts of logs.o.o? so that URLs like http://logs.openstack.org/54/644554/2/check/tempest-full-py3/42292da/job-output.txt.gz don't get redownloaded by a client every time they are viewed | 11:17 |
| aspiers | currently Apache doesn't return any header about when it was last modified | 11:17 |
| aspiers | does the 42292da bit change on a recheck? | 11:18 |
| *** jtomasek has joined #openstack-infra | 11:23 | |
| *** ccamacho has quit IRC | 11:24 | |
| *** ccamacho has joined #openstack-infra | 11:26 | |
| *** rcernin has joined #openstack-infra | 11:27 | |
| *** njohnston has joined #openstack-infra | 11:27 | |
| *** udesale has quit IRC | 11:28 | |
| *** udesale has joined #openstack-infra | 11:29 | |
| *** lpetrut has joined #openstack-infra | 11:29 | |
| *** pcaruana has quit IRC | 11:38 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** yamamoto has joined #openstack-infra | 11:41 | |
| *** jpich has quit IRC | 11:42 | |
| *** jpich has joined #openstack-infra | 11:42 | |
| tosky | the bump of ansible raised the requirements of openstacksdk for the os_user module, and now I have a job failing on stable/queens: http://logs.openstack.org/83/643383/2/check/sahara-tests-scenario-queens/a6fc707/job-output.txt.gz#_2019-03-20_11_37_50_303552 | 11:45 |
| tosky | what can I do? Not use os_user? | 11:46 |
| *** yamamoto has quit IRC | 11:46 | |
| *** jpich has quit IRC | 11:49 | |
| *** jpich has joined #openstack-infra | 11:49 | |
| *** dave-mccowan has quit IRC | 11:51 | |
| *** rh-jelabarre has joined #openstack-infra | 11:52 | |
| cdent | It appears that placement-stable-maint was created but we failed to give it any members: https://review.openstack.org/#/admin/groups/1938,members Could someone please stick stable-maint-core in there, plus me. Please and thank you. I'd like to merge the stable/stein patches but can't | 11:56 |
| *** e0ne has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:10 | |
| *** jbadiapa has quit IRC | 12:11 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** psachin has quit IRC | 12:11 | |
| *** markvoelker has quit IRC | 12:13 | |
| *** iury|lunch is now known as iurygregory | 12:14 | |
| *** pcaruana has joined #openstack-infra | 12:16 | |
| frickler | cdent: I added you, which I guess as fresh PTL should be fine, you should now be able to self-manage other members/groups | 12:16 |
| cdent | great, thanks very much | 12:16 |
| aspiers | cdent: I added the traits topic to the Denver placement etherpad but not sure if that's the right forum for it | 12:17 |
| cdent | aspiers: yeah, I left a response on the etherpad with some ideas on how to decide | 12:18 |
| *** psachin has joined #openstack-infra | 12:19 | |
| *** jpich has quit IRC | 12:19 | |
| aspiers | cdent: Thanks, I should have guessed you'd already be ahead of me ;-) | 12:19 |
| frickler | tosky: hmm, I was thinking that the ansible setup should be encapsuled in a venv. if that isn't the case and the needed sdk version is too high for queens, we might want to stick to using older ansible for stable branches. maybe other infra-root have better ideas | 12:19 |
| *** jpich has joined #openstack-infra | 12:20 | |
| cdent | aspiers: it's not always the case... | 12:20 |
| tosky | frickler: it does not seem to be the case (venv) | 12:21 |
| tosky | pike is affected too, but pike jobs which uses advanced zuul v3 features are probably rarer, and more important, pike is moving to Extended Maintenance | 12:21 |
| frickler | tosky: maybe we just lack installing it into the venv, there was something that sounded similar to me about gear. I can check later | 12:22 |
| frickler | corvus: please see tosky's error above, I'm thinking that we would also want to add openstacksdk here, though I'm not 100% sure that that's the correct location https://review.openstack.org/#/c/644685/1/manifests/executor.pp | 12:26 |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** rlandy has joined #openstack-infra | 12:31 | |
| *** jpich has quit IRC | 12:35 | |
| *** jpich has joined #openstack-infra | 12:35 | |
| *** gfidente has joined #openstack-infra | 12:39 | |
| frickler | infra-root: this looks similar to the tempest issue earlier, seems another fallout of the ansible update. though I haven't understood yet completely at which location the handling of undefined variables has been changed https://bugs.launchpad.net/neutron/+bug/1820993 | 12:39 |
| openstack | Launchpad bug 1820993 in neutron "Functional and fullstack tests job definition is broken" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq) | 12:39 |
| *** jpich has quit IRC | 12:39 | |
| *** njohnston has quit IRC | 12:40 | |
| *** njohnston has joined #openstack-infra | 12:41 | |
| *** eharney has joined #openstack-infra | 12:42 | |
| *** kgiusti has joined #openstack-infra | 12:47 | |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| frickler | although actually this is a different issue it seems | 12:50 |
| *** cdent has left #openstack-infra | 12:51 | |
| *** jaosorior has quit IRC | 12:56 | |
| fungi | frickler: ansible is installed into per-version virtualenvs on the executors, and the gear import errors yesterday were because we neglected to install gear into those virtualenvs | 12:56 |
| *** hwoarang has quit IRC | 13:00 | |
| fungi | i guess in the devstack jobs ansible is also getting installed globally on the "controller" node? | 13:01 |
| *** hwoarang has joined #openstack-infra | 13:01 | |
| frickler | fungi: is it? or does ansible just copy things when executing tasks on other hosts? | 13:02 |
| fungi | yeah, maybe... i'm still a bit fuzzy on ansible's remote execution module... is the idea that it's importing the openstacksdk module installed in a system context on the controller node? | 13:04 |
| fungi | i guess it copies some scripts over ssh and then runs those with the local python, so would use the modules installed in the system context on the remote host | 13:04 |
| tosky | fungi: from the log above I see that openstacksdk 0.11.3 is installed on the controller node (it's a single-node job) and os_user, executed on that node, complains about the version of openstacksdk | 13:05 |
| fungi | yeah, hunting around, the solutions for using a virtualenv on the remote host during execution are to have one task create a virtualenv on the remote using the python from the system context, and then for subsequent operations set ansible_python_interpreter to the path of the interpreter in the remote virtualenv | 13:07 |
| fungi | so i guess there's not a simple switch to throw for isolating the remote ansible operations from the remote host's system context modules | 13:08 |
| tosky | but should that be explicitly done every time I want to use os_user or other openstacksdk-based modules on a remote node? | 13:08 |
| tosky | I'd rather have it working transparently | 13:09 |
| *** jpich has joined #openstack-infra | 13:09 | |
| fungi | tosky: oh, i'm nowhere near making recommendations yet, still just trying to wrap my head around how ansible works | 13:09 |
| tosky | oki | 13:09 |
| *** lseki has joined #openstack-infra | 13:09 | |
| Shrews | it should be safe to always use the latest version of openstacksdk, no matter the version of ansible. possibly just have a task to install the latest version before using the os_* modules? | 13:10 |
| tosky | Shrews: if it's safe, why does upper-constraints from requirements.txt limit it? | 13:10 |
| fungi | Shrews: that seems like one option... i guess the problem is if devstack wants to enforce that we test with the sdk version in stable constraints | 13:10 |
| fungi | and its constrained dependencies | 13:11 |
| tosky | because it's installed system-wide, and it may impact other bits | 13:11 |
| Shrews | fungi: yeah. if you could isolate the sdk version for just the ansible context, that would be ideal | 13:11 |
| fungi | this gets back to the challenges inherent in using python-based tools to test python-based software | 13:12 |
| fungi | it's possible we may want to tie the ansible version for jobs to the same job variants we're using to decide which versions of our platforms we use | 13:12 |
| fungi | so keep stable/rocky and earlier jobs running with ansible 2.5, and use 2.7 on stable/stein and master? | 13:13 |
| *** ramishra has quit IRC | 13:14 | |
| frickler | fungi: if it's not possible to untangle things with a venv easily, that may indeed be the best solution | 13:14 |
| fungi | or, yeah, find a way to isolate the remote python environment used by ansible from the remote python environment used by software being tested... but then you can't easily share the same installations of some dependencies between the job definitions and the software being tested | 13:15 |
| *** rcernin has quit IRC | 13:16 | |
| Shrews | fungi: jobs can selectively choose the ansible version, right? | 13:16 |
| *** e0ne has joined #openstack-infra | 13:16 | |
| fungi | yes | 13:16 |
| fungi | which is why i was suggesting maybe we select the ansible version for specific stable branches | 13:17 |
| *** mriedem has joined #openstack-infra | 13:17 | |
| fungi | as it's possible some of them will be running jobs which need ansible dependencies contemporary with when the constraints were frozen for those branches | 13:17 |
| Shrews | so automatically do it on a global scale? | 13:18 |
| fungi | either that or continue to recommend one-off fixes for every job which uses openstacksdk on a stable branch | 13:18 |
| fungi | (and whatever other modules have similar challenges we haven't found out about yet) | 13:19 |
| Shrews | the former actually makes sense | 13:19 |
| fungi | the *hope* is that stable branches should mostly be a state frozen in time and not need changing due to outside alterations of the ci system | 13:19 |
| pabelanger | frickler: yah, 644842 would silent warning the syntax wasn't valid, in 2.7 ansible will error. So fix here is to use valid syntax | 13:19 |
| pabelanger | (2.5 would silent warn) | 13:20 |
| Shrews | that ^ is also another good reason for ansible versioning by branch | 13:21 |
| fungi | but apparently the ansible version used on the executor is not necessarily independent of the versions of dependencies needed by its ansible plugins on remote hosts | 13:21 |
| *** jbadiapa has joined #openstack-infra | 13:21 | |
| frickler | pabelanger: would 2.6 give a visible warning? | 13:21 |
| *** jaosorior has joined #openstack-infra | 13:21 | |
| frickler | pabelanger: or is there a way to see the warning with 2.5? we might want to crosscheck for other possibly affected jobs | 13:22 |
| *** ccamacho has quit IRC | 13:23 | |
| *** ccamacho has joined #openstack-infra | 13:23 | |
| pabelanger | frickler: I suspect you'd see something like [WARNING]: Ignoring invalid attribute: foo | 13:25 |
| pabelanger | frickler: should be able to use logstash... let me look | 13:25 |
| pabelanger | ah, yah. I see things in logstash | 13:26 |
| pabelanger | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%5BWARNING%5D%3A%20Ignoring%20invalid%20attribute%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1 | 13:26 |
| *** ccamacho has quit IRC | 13:28 | |
| *** ccamacho has joined #openstack-infra | 13:28 | |
| *** psachin has quit IRC | 13:30 | |
| *** njohnston has quit IRC | 13:30 | |
| *** njohnston has joined #openstack-infra | 13:31 | |
| jrosser | fwiw we take great care over ansible versioning per branch in openstack-ansible, you might be able to recycle some of that | 13:32 |
| frickler | pabelanger: woa, that's a lot of matches. but these still seem to be non-fatal even now, is that correct? | 13:32 |
| jrosser | all of those issues with os_* modules are very familiar | 13:33 |
| *** sthussey has joined #openstack-infra | 13:34 | |
| pabelanger | frickler: it is possible these are also nested ansible, I haven't looked too much | 13:36 |
| pabelanger | but talking with ansible-core, it is possible some are still warning, and some now error | 13:36 |
| *** njohnston has quit IRC | 13:36 | |
| pabelanger | there is work to clean up attribute parsing on tasks | 13:36 |
| pabelanger | so, a work item should be to fix all the warnings | 13:37 |
| *** jpena|lunch is now known as jpena | 13:39 | |
| frickler | pabelanger: I still fail to find a warning with "invalid attribute: msg" that would match the neutron failure | 13:45 |
| pabelanger | frickler: I wonder if that is showing up in the executor log | 13:45 |
| pabelanger | since that is where ansible is running | 13:45 |
| pabelanger | let me look real quick | 13:46 |
| frickler | pabelanger: executor-debug.log, indeed | 13:47 |
| *** helenafm has quit IRC | 13:48 | |
| *** jonher has quit IRC | 13:54 | |
| *** ramishra has joined #openstack-infra | 13:54 | |
| *** ykarel is now known as ykarel|afk | 13:56 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add job and integration tests for the upload-git-mirror role https://review.openstack.org/643435 | 13:59 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add job and integration tests for the upload-git-mirror role https://review.openstack.org/643435 | 14:00 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to mirror a git repository to a remote git server https://review.openstack.org/643394 | 14:00 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Do not merge: test zuul-jobs-upload-git-mirror job https://review.openstack.org/643437 | 14:00 |
| *** jonher has joined #openstack-infra | 14:02 | |
| *** eharney has quit IRC | 14:03 | |
| *** ykarel|afk is now known as ykarel | 14:08 | |
| pabelanger | frickler: INVALID_TASK_ATTRIBUTE_FAILED | 14:08 |
| pabelanger | that should restore the original (broken) tasks | 14:08 |
| pabelanger | in 2.7 that was flipped to true, which is breaking everything here | 14:08 |
| pabelanger | well, not everything, but failing on the invalid tasks so far | 14:09 |
| pabelanger | frickler: also: http://logs.openstack.org/85/643485/5/check/neutron-functional/e06a057/job-output.txt.gz#_2019-03-20_08_37_50_699476 was pointed out by human in ansible | 14:11 |
| fungi | #status log restarted mysql and apache2 services on storyboard01.opendev.org to investigate cache memory pressure | 14:20 |
| openstackstatus | fungi: finished logging | 14:20 |
| *** jamesmcarthur has quit IRC | 14:25 | |
| *** jamesmcarthur has joined #openstack-infra | 14:25 | |
| *** ykarel is now known as ykarel|afk | 14:26 | |
| *** chandankumar is now known as raukadah | 14:29 | |
| openstackgerrit | jacky06 proposed openstack/os-testr master: Update hacking version https://review.openstack.org/641384 | 14:30 |
| *** derekh has quit IRC | 14:31 | |
| clarkb | the issue is anaible 2.7 tries to use job installed sdk libs and fails because it is too old? | 14:35 |
| *** ykarel_ has joined #openstack-infra | 14:35 | |
| clarkb | aspiers: the access patterns for those artifacts havent really justified it in the past. Most are single access by our log processong tools | 14:36 |
| *** ykarel|afk has quit IRC | 14:37 | |
| fungi | clarkb: correct | 14:41 |
| *** electrofelix has quit IRC | 14:41 | |
| fungi | clarkb: in short, devstack-managed services want some of the same libraries as ansible plugnis executed remotely on the job nodes, but disagree on what versions they need | 14:41 |
| fungi | and since the devstack jobs in particular seem to want to use the openstack plugin for ansible they particularly collide over expected openstacksdk version | 14:42 |
| clarkb | I wonderif sdk is added as an extrlib if ansible will vendor it to the remote host | 14:43 |
| frickler | it's also not devstack jobs in general, but some sahara job using the os_user module, iiuc | 14:44 |
| clarkb | but also seems like a good reason to not use ansible for that and instead rely on remote scripts | 14:44 |
| clarkb | eg take this as a lesson that this doesnt work long term and avoid doing it | 14:44 |
| fungi | yeah, i'm less concerned about this specific case, but more the portent that upgrading the version of ansible we use to run jobs can wind up at odds with libraries shared by the remote ansible scripts and the software on the remote nodes | 14:45 |
| fungi | and the older the branches being tested, the greater that risk probably | 14:46 |
| frickler | this is the role definition, do we really want to enforce people using osc shellouts like the one below everywhere? https://opendev.org/openstack/sahara-tests/src/branch/master/roles/setup-sahara-scenario-env/tasks/main.yaml#L16-L22 | 14:49 |
| fungi | we can recommend against it, and i like the vendoring idea too, just wondering how we detect that before it ends up baked into a stable branch for two years with no problem and suddenly no longer works after an ansible upgrade | 14:49 |
| frickler | s/enforce/persuade/ or whatever | 14:49 |
| *** dtantsur is now known as dtantsur|bbl | 14:51 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for gabbi_tempest_path bug 1821007 https://review.openstack.org/644885 | 14:51 |
| openstack | bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] https://launchpad.net/bugs/1821007 - Assigned to Paul Belanger (pabelanger) | 14:51 |
| *** jbadiapa has quit IRC | 14:51 | |
| clarkb | frickler: if we are pinning the install of that toolchain in that job then ya I think we have to | 14:52 |
| clarkb | fungi: this class of problem is why historically openstack hasn't tried to support stable branches for very long :) | 14:53 |
| fungi | yup | 14:53 |
| clarkb | worth noting that my now mostly abandoned plan to put everything into a virtualenv with devstack would've addressed this by forcing the job to manage the ansible sdk version indepednent of the job sdk version | 14:54 |
| frickler | clarkb: iiuc we don't install the openstacksdk needed for ansible at all. devstack happens to do so globally by chance, but on stable/queens with a version that is too old for ansible 2.7 | 14:54 |
| fungi | clarkb: however the last thing i want is for the support limitations to be blamed on our infrastructure | 14:54 |
| clarkb | frickler: ya, I think this poinst to further benefit to getting everything into a devstack venv for devstack | 14:55 |
| clarkb | its just that happens to be another huage change that will break even more stuff | 14:55 |
| fungi | yeah, and also devstack-in-a-venv probably doesn't help the existing stable branches which are unlikely to get ported to it | 14:55 |
| fungi | still, might be worth revisiting in the u cycle when devstack/openstack no longer needs to care about python 2.7 | 14:56 |
| fungi | or really any python <3.6 | 14:57 |
| *** ykarel_ is now known as ykarel | 14:58 | |
| dmsimard | Did anyone have comments about this github org transfer script I wrote ? Should I put it to a repo somewhere ? See: https://gist.github.com/dmsimard/7767f20f3a03ab26685996ccc9ee6493 | 14:58 |
| fungi | unfortunately centos 8 will likely be 3.6 for a very long time and even the latest ubuntu lts probably won't default to newer than 3.6 until 20.04 (which is presumably not relevant for openstack until the v cycle) | 14:59 |
| corvus | dmsimard: it's confusing to put "all rights reserved" in an apache license header. | 14:59 |
| fungi | dmsimard: propose to openstack-infra/system-config tools directory maybe? | 14:59 |
| corvus | fungi: ++ | 14:59 |
| dmsimard | corvus: copy pasta from somewhere else, I can remove it | 15:00 |
| fungi | and i agree, "all rights reserved" is neither useful nor relevant | 15:00 |
| dmsimard | I'll take note for the future, I'm not good with that stuff. | 15:00 |
| fungi | i think that phrase ceased being relevant for any works as of the berne convention? | 15:00 |
| dmsimard | fungi: ack, I'll send it there | 15:00 |
| *** eggshell has joined #openstack-infra | 15:00 | |
| *** lpetrut has quit IRC | 15:01 | |
| fungi | but in particular as corvus points out, it also runs counter to the intent of free/libre open source software licenses | 15:01 |
| * dmsimard finds a proper license block header | 15:01 | |
| *** eggshell has left #openstack-infra | 15:02 | |
| pabelanger | I had a question with promote pipeline for openstack, if a job was both in promote / post, does zuul basically enqueue them at the same time? Given they are 2 different event triggers from gerrit? | 15:02 |
| *** manjeets_ is now known as manjeets | 15:02 | |
| clarkb | pabelanger: they should end up in the queues at the same time since zuul will process the triggering event once (then it iterates through it tspipelines to find matches) but the jobs will not run in any specific order depending on nodepool and zuul executor availability | 15:04 |
| corvus | fungi, frickler, clarkb: it seems that, on a high level, the sahara job was written to use a certain version of ansible (whether its author intended that or not). so an appropriate fix would be to pin it to that version of ansible. another fix would be to alter it to support more versions by avoiding the dependency on sdk. | 15:04 |
| corvus | fungi, frickler, clarkb: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.ansible-version is how to set the ansible version | 15:04 |
| fungi | dmsimard: see the end of https://www.apache.org/dev/apply-license.html | 15:04 |
| fungi | where it says "APPENDIX: How to apply the Apache License to your work. | 15:04 |
| fungi | " | 15:04 |
| corvus | fungi, frickler, clarkb: does ansible 2.6 work? if so, that might extend the life of the job as written for some time. | 15:04 |
| fungi | dmsimard: er, i pasted the wrong url, meant https://www.apache.org/licenses/LICENSE-2.0.txt | 15:05 |
| pabelanger | clarkb: thanks, that is what I figured. I was mostly curious what differences the ref-updated and change-merged trigger events provided in the 2 pipelines. Because I don't think github has the same granularity as gerrit in this case | 15:06 |
| corvus | pabelanger: the jobs needed for post and promote are usually different enough that it doesn't make sense to put them in both... | 15:06 |
| *** dpawlik is now known as dpawlik_ | 15:06 | |
| corvus | pabelanger: the differences are almost exactly the same in gerrit and github | 15:06 |
| frickler | tosky: ^^ can you try to pin your job to ansible 2.6 or maybe 2.5? | 15:09 |
| *** hwoarang has quit IRC | 15:09 | |
| pabelanger | corvus: in zuul.o.o case, could the trigger event on promote (not that I am asking for it to change) be the same as post? Or is there specific ordering in gerrit that change-merge / ref-updated provide? | 15:09 |
| *** hwoarang has joined #openstack-infra | 15:10 | |
| pabelanger | corvus: in the case of github pipeline, I think I can only use the pushevent, which is what is setup for post today. The only difference between the stanza would be, promote would comment on the change (PR), which is really what I am looking for | 15:10 |
| corvus | pabelanger: the trigger for post has the merge commit sha but not the change that caused it. therefore post is used for building artifacts that must be built from the final sha. promote has a reference to a merged change, but not the final merge commit sha, so it's appropriate for dealing with artifacts created for changes in gate, but where we don't need the assurance that they were actually | 15:11 |
| corvus | built from the final git sha. | 15:11 |
| openstackgerrit | Rico Lin proposed openstack-infra/irc-meetings master: Change Heat meeting time https://review.openstack.org/644896 | 15:11 |
| pabelanger | corvus: Great, that is info I was missing! thankyou | 15:12 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for gabbi_tempest_path bug 1821007 https://review.openstack.org/644885 | 15:13 |
| openstack | bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] https://launchpad.net/bugs/1821007 - Assigned to Paul Belanger (pabelanger) | 15:13 |
| *** yamamoto has quit IRC | 15:16 | |
| *** yamamoto has joined #openstack-infra | 15:18 | |
| *** yamamoto has quit IRC | 15:18 | |
| *** yamamoto has joined #openstack-infra | 15:18 | |
| *** yamamoto has quit IRC | 15:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Forward artifacts to child jobs within buildset https://review.openstack.org/642857 | 15:26 |
| asettle | Hey infra team - did anyone/does anyone have any documentation on how we implemented the building of openstack docs as PDFs? ianychoi AJaeger et al | 15:27 |
| clarkb | asettle: I don't know off the top of my head, but my guess is via sphinx as it can do pdf output iirc (as well as html and manpage/rof format) | 15:28 |
| asettle | How did we handle the cross refs with the build PDF? Did we just comment them out? | 15:29 |
| *** altlogbot_1 has joined #openstack-infra | 15:29 | |
| *** altlogbot_1 has quit IRC | 15:30 | |
| mordred | asettle: I don't think we comment them out - I think sphinx just DTRT | 15:35 |
| asettle | That's good to know. I don't know why but I have this huge image in my head that it was much, much more complicated than simply implemetning the function. | 15:36 |
| asettle | implementing * | 15:36 |
| corvus | asettle: pdfs support both internal and external hyperlinks, so both kinds of references should work | 15:37 |
| asettle | Oh neat, even better. Thanks! | 15:38 |
| asettle | So there's no infra trick of the light I'm missing here? we implemented the thing, the thing worked, we put it across the repos, badda bing, badda boom? | 15:38 |
| fungi | that matches my (vague) recollection, yes | 15:39 |
| asettle | Perfecto | 15:40 |
| *** eharney has joined #openstack-infra | 15:40 | |
| asettle | Thanks team :D | 15:40 |
| mordred | asettle: the hardest bit was figuring out the bindep depends for it | 15:48 |
| asettle | Ah. That's not so bad. | 15:48 |
| asettle | (for me) | 15:48 |
| AJaeger | asettle: we're not building PDFs in general, only for openstack-manuals | 15:48 |
| asettle | AJaeger, yep! | 15:49 |
| mordred | asettle: there's a couple of different options, and one works while others don't. I don't remember which were the right ones | 15:49 |
| asettle | Is it possible to build a PDF of say, the nova admin guide? | 15:49 |
| mordred | asettle: but - looking through openstack-manuals for what the bindeps are is probably a good choice | 15:49 |
| AJaeger | asettle: and there it's only tox - and bindep.txt has the binary requirements | 15:49 |
| asettle | Good call mordred | 15:49 |
| AJaeger | asettle: yes, update doc/source/conf.py similar like done in openstack-manuals/doc/*/source/conf.py... | 15:49 |
| mordred | asettle: I don't think there is anything that would cause that to not be possible - other than a general decision about doing so. and what AJaeger said :) | 15:49 |
| * AJaeger is travelling, will go offline again | 15:50 | |
| asettle | AJaeger, o/ thank you | 15:50 |
| asettle | mordred, so currently not implemented, but doable | 15:51 |
| *** gyee has joined #openstack-infra | 15:51 | |
| *** ykarel is now known as ykarel|away | 15:51 | |
| *** lpetrut has joined #openstack-infra | 15:54 | |
| fungi | as a heads up, looks like https://review.openstack.org/644842 is more fallout from the ansible upgrade | 15:55 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 15:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add retries to promote-docker-image https://review.openstack.org/644911 | 16:02 |
| *** Lucas_Gray has quit IRC | 16:03 | |
| *** efried is now known as efried_rolling | 16:03 | |
| openstackgerrit | Daniel Abad proposed openstack/diskimage-builder master: Minor clarifications in centos7 element docs https://review.openstack.org/644913 | 16:04 |
| fungi | gonna go find lunch and run a quick errand, but should be back in an hour or so | 16:05 |
| openstackgerrit | Daniel Abad proposed openstack/diskimage-builder master: Minor clarifications in centos7 element docs https://review.openstack.org/644913 | 16:05 |
| *** lpetrut has quit IRC | 16:12 | |
| *** rfolco has quit IRC | 16:14 | |
| *** rfolco has joined #openstack-infra | 16:14 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Whitelist `rm` for docs target in tox.ini https://review.openstack.org/644920 | 16:15 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: WIP: Add docker-compose based dev setup instructions https://review.openstack.org/644921 | 16:15 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: WIP: rewrite/clarify non-docker dev env instructions https://review.openstack.org/644922 | 16:15 |
| *** rfolco has quit IRC | 16:16 | |
| *** rfolco has joined #openstack-infra | 16:17 | |
| corvus | fungi, clarkb, frickler, pabelanger: how's this look? https://etherpad.openstack.org/p/dihydGYxsm | 16:19 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: wip - Elasticsearch Zuul reporter https://review.openstack.org/644927 | 16:22 |
| clarkb | corvus: re the default being 2.7 is that the current value of a default being latest ansible release? or are we going to be more discerning about that? | 16:22 |
| *** jbadiapa has joined #openstack-infra | 16:22 | |
| pabelanger | corvus: wfm! | 16:22 |
| *** dtantsur|bbl is now known as dtantsur | 16:22 | |
| clarkb | corvus: also links to the docs on setting ansible version per job will likely help readers | 16:23 |
| clarkb | otherwise lgtm | 16:23 |
| tosky | corvus: the sahara job was written to use a well-documented feature of ansible, the os_user task; if zuul provides "ansible" to write tasks, I shouldn't care how the dependencies are working, I think | 16:23 |
| *** sshnaidm|afk is now known as sshnaidm|rover | 16:23 | |
| openstackgerrit | Daniel Abad proposed openstack/diskimage-builder master: Minor clarifications in centos7 element docs https://review.openstack.org/644913 | 16:23 |
| tosky | frickler: how do I pin my job to ansible 2.5? (I think that 2.6 has the same issue) | 16:24 |
| corvus | clarkb: i think our policy is going to be latest ansible? or maybe latest after a very short stabilization delay? i expect we may want to add support for pre-releases in the future though, so at some point i expect people to be able to select future versions. | 16:24 |
| clarkb | tosky: unfortunately that isn't how ansible works | 16:24 |
| clarkb | tosky: if you'd like to change that you'll need to change how nasible works aiui (which may be a valid bug from their perspective) | 16:24 |
| clarkb | tosky: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.ansible-version | 16:24 |
| tosky | clarkb: maybe that's more how openstacksdk works | 16:24 |
| corvus | tosky: no it's how zuul works -- did you read my explanation of item 2 in https://etherpad.openstack.org/p/dihydGYxsm ? | 16:25 |
| corvus | tosky: that's true for all ansible modules | 16:25 |
| clarkb | corvus: ya so we may want to say "default is going to be latest ansible release which is currently 2.7" | 16:25 |
| corvus | tosky: er, rather, it's how *ansible* works | 16:25 |
| clarkb | corvus: that way when it becomes 2.8 people don'e complain we said 2.7 is the default | 16:25 |
| *** trown is now known as trown|lunch | 16:25 | |
| tosky | I suspect that these kind of issues will disappear when the modules will stabilize | 16:26 |
| corvus | tosky: i hope so | 16:26 |
| *** mattw4 has joined #openstack-infra | 16:26 | |
| tosky | corvus: but anyway, it may happen again now that the number of native zuul v3 jobs is increasing, and given the short lifetime of ansible and the longer lifetime of openstack | 16:28 |
| clarkb | corvus: edits lgtm on etherpad | 16:28 |
| *** ramishra has quit IRC | 16:28 | |
| corvus | tosky: yeah, when mordred is back from his trip, we should chat about this, because i don't think this issue was on his radar :) | 16:29 |
| tosky | this specific job is a defined in a branchless repository , which means that every job defined in a branchless repository should not use potentially problematic features | 16:30 |
| corvus | email sent | 16:32 |
| tosky | if I read that email correctly, it means that I can't even switch on ansible 2.5, because that will be removed soon | 16:32 |
| tosky | when 2.8 is out, according the ansible support policy | 16:32 |
| tosky | (last 3 releases) | 16:33 |
| corvus | tosky: you can switch to 2.5, and it will continue working until at least the beginning of May | 16:33 |
| tosky | and I suspect that it will happen long before queens turns into Extended Maintenance | 16:33 |
| *** priteau has joined #openstack-infra | 16:33 | |
| corvus | but it's correct that is not a long-term solution | 16:33 |
| corvus | tosky: did you verify that 2.6 also requires a newer sdk? | 16:35 |
| *** mattw4 has quit IRC | 16:36 | |
| tosky | corvus: https://github.com/ansible/ansible/issues/42588 | 16:36 |
| *** altlogbot_1 has joined #openstack-infra | 16:39 | |
| corvus | tosky: :( then i'd suggest setting the version to 2.5 for now, and if mordred or Shrews don't have any other suggestions over the next few days, i guess convert those os_* tasks to command | 16:41 |
| Shrews | if ansible had to install every external library needed for all of its included modules... oy, the problems that could cause | 16:41 |
| *** altlogbot_1 has quit IRC | 16:41 | |
| corvus | Shrews: i think it would be theoretically possible for it to install that into some kind of private venv on the remote side. but that's obviously some substantial engineering. | 16:42 |
| Shrews | "install the latest sdk" has always been our standard response. the problem is that as features are added to openstack API, the modules have necessary reflected those features. thus we add the updated sdk requirement | 16:43 |
| corvus | i did have the thought that some of the os_* tasks could be run on the executor instead of the "controller" node, but i think the task we were specifically looking at earlier was an image upload? so that would probably not be a good fit since it would involve shipping the image back to the executor. | 16:43 |
| corvus | Shrews: right, it seems that stance is at odds with the use case "use the ansible openstack module in a zuul job on openstack's stable branches" | 16:44 |
| Shrews | yeah | 16:44 |
| corvus | Shrews, tosky, clarkb: would this be a potential solution? https://stackoverflow.com/questions/20575084/best-way-to-always-run-ansible-inside-a-virtualenv-on-remote-machines | 16:49 |
| *** yolanda has quit IRC | 16:49 | |
| Shrews | we used to have the modules restrict min version based on the feature being used. at some point we moved it to a common code base so that it's now per-module. that may have been a mistake in hindsight | 16:49 |
| *** mattw4 has joined #openstack-infra | 16:51 | |
| Shrews | corvus: i mean, any process that would separate the ansible env from the one needed by devstack should work. so i think yeah, that's an option | 16:52 |
| *** jamesmcarthur has quit IRC | 16:52 | |
| *** armax has joined #openstack-infra | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:53 | |
| Shrews | i've run ansible that way myself for some reason i've now forgotten | 16:54 |
| clarkb | If we think about people using these tools in productuin they will have the same issues too | 16:55 |
| clarkb | definitely seems worthwhile to improve in ansible if possible given that | 16:55 |
| Shrews | oh, heh... issue #2: https://dshrewsbury.blogspot.com/2016/02/common-problems-using-ansible-20.html | 16:55 |
| corvus | we could actually pre-create a virtualenv in nodepool images and then jobs which want to use it could set ansible_python_interpreter | 16:56 |
| Shrews | corvus: or do something in the base jobs to prepare it maybe? | 16:58 |
| *** mattw4 has quit IRC | 16:58 | |
| corvus | Shrews: yes, either way... the usual complexity/time trade-off :) | 17:00 |
| *** mattw4 has joined #openstack-infra | 17:03 | |
| *** ykarel|away has quit IRC | 17:04 | |
| *** mattw4 has quit IRC | 17:05 | |
| *** jbadiapa has quit IRC | 17:07 | |
| *** e0ne has quit IRC | 17:07 | |
| *** ykarel has joined #openstack-infra | 17:08 | |
| *** mattw4 has joined #openstack-infra | 17:09 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add script to automate GitHub organization transfers https://review.openstack.org/644937 | 17:10 |
| *** ykarel is now known as ykarel|pto | 17:10 | |
| corvus | tosky: ^ what do you think about that? should we set up a virtualenv on the remote node for zuul's ansible? i think that means we can keep zuul's ansible modules isolated from devstack and the jobs would not need to change. or would you rather rework the jobs to not use os_ modules? | 17:11 |
| *** yamamoto has joined #openstack-infra | 17:11 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Typo fix: s/extract_mesages/extract_messages/ https://review.openstack.org/644942 | 17:16 |
| *** udesale has quit IRC | 17:17 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Fix nits and typos on release note message. https://review.openstack.org/644563 | 17:19 |
| *** trown|lunch is now known as trown | 17:20 | |
| *** altlogbot_1 has joined #openstack-infra | 17:24 | |
| *** altlogbot_1 has quit IRC | 17:24 | |
| *** altlogbot_0 has joined #openstack-infra | 17:26 | |
| clarkb | corvus: the jobs would need to install the tools in the venv? or are you thinking bake in some common tools? | 17:27 |
| *** yamamoto has quit IRC | 17:27 | |
| *** altlogbot_0 has quit IRC | 17:28 | |
| *** armax has quit IRC | 17:28 | |
| corvus | clarkb: either would work i think; we could provide an empty venv that jobs can opt into and start pip installing, or we could go ahead and bake openstacksdk in. | 17:28 |
| clarkb | thinking about it more baking in a small number of useful tools for ansible is probably a reasonable thing to do | 17:29 |
| clarkb | the openshift client and kubernetes sdk too | 17:29 |
| *** altlogbot_3 has joined #openstack-infra | 17:29 | |
| *** yamamoto has joined #openstack-infra | 17:30 | |
| *** yamamoto has quit IRC | 17:30 | |
| *** yamamoto has joined #openstack-infra | 17:30 | |
| *** raissa has joined #openstack-infra | 17:30 | |
| clarkb | infra-root (mordred in particular as you wrote the breaking change) https://review.openstack.org/#/c/644432/ is something I'd like to test on review-dev today. Assuming that goes well I think we should try and get that in and then run manage-projects by hand on prod gerrit (run by hand because it will likely be slow as it fixes everything and go past the puppet timeout) | 17:33 |
| mordred | corvus, Shrews: I'm not 100% caught up on the backscroll - I guess the reason we can't install master sdk is that we'd be polluting the test? | 17:33 |
| clarkb | some quick reviews on that before I start running things would be appreciated just to make sure I'm not missing something obvious and in the meantime I need to debug those openstackiddev node link loops | 17:33 |
| *** jpich has quit IRC | 17:33 | |
| clarkb | mordred: right openstack sets an upper constraint for the branch that is older than master | 17:33 |
| mordred | AH | 17:34 |
| clarkb | mordred: and openstack doesn't like bumping upper constraints on stable branches | 17:34 |
| mordred | yeah | 17:34 |
| *** ginopc has quit IRC | 17:34 | |
| *** yamamoto has quit IRC | 17:35 | |
| mordred | is there a change I can look at that is being affected? | 17:35 |
| clarkb | mordred: http://logs.openstack.org/83/643383/2/check/sahara-tests-scenario-queens/a6fc707/job-output.txt.gz#_2019-03-20_11_37_50_303552 is the job and 643383 is the change I think | 17:35 |
| mordred | I would expect, possibly incorrectly, that if things aren't trying to use new 2.7 features of os_user - that old sdk should still work | 17:36 |
| mordred | clarkb: thanks. looking now | 17:36 |
| *** ccamacho has quit IRC | 17:36 | |
| clarkb | mordred: shrews mentioned that os_* used to have min versions for each os_* thing but then there as a module combining and now it is global to all os_* things? | 17:36 |
| Shrews | clarkb: it was moved from module level to https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/openstack.py#L111 | 17:38 |
| mordred | yeah | 17:40 |
| tosky | corvus: creating the venv is a bigger change, I guess, and probably more invasive; I will probably remove the usage of os_user for now, but if we had a clear pattern it would be better, I think | 17:40 |
| mordred | I think we should set ansible_version to 2.5 for the stable/queens version of the job | 17:42 |
| mordred | the min version that's set is the min version of when the shade->sdk transition happened, it's not shifting upwards each release | 17:42 |
| * fungi is back now | 17:43 | |
| fungi | corvus: the draft announcement lgtm, thanks! | 17:43 |
| mordred | so it should really only be a problem with stable/queens since that was before the good version of sdk | 17:43 |
| *** ykarel|pto has quit IRC | 17:43 | |
| *** ricolin has quit IRC | 17:44 | |
| mordred | 2.7 should work with all the other stable branches newer than queens | 17:44 |
| mordred | tosky: ^^ sorry, I should have tagged you in that | 17:44 |
| mordred | or, zuul_ansible_version that is | 17:45 |
| tosky | mordred: yep, it does work with rocky | 17:45 |
| tosky | mordred: the problem is that 2.5 is probably going out of support before the end of life of queens | 17:45 |
| mordred | ah. well poop | 17:45 |
| clarkb | we could in theory still support eol ansible in jobs (just recommend against it | 17:46 |
| mordred | sorry - that's a really unfortunate combination of circumstances | 17:46 |
| clarkb | and if it doesn't work then we aren't doing much to fix it | 17:46 |
| tosky | I'm going to remove the offending code, but it may happen again | 17:47 |
| mordred | tosky: this shouldn't be an issue post-queens - unfortunately rocky was the transition time - and we didn't think to map 2.5 lifecycle to queens lifecycle to consider this | 17:48 |
| fungi | also worth remembering, openstack decided to stop just ending live of stable branches, so stable/queens is expected to continue accepting patches long after it stops getting point releases. this will be a fun conundrum for the extended maintenance team to unwind | 17:51 |
| clarkb | fungi: yup, but it is also contingent on being able to keep things working. I think not using the os_* modules on queens and older is a reasonable path there for that | 17:52 |
| fungi | sure, that's certainly one way out | 17:53 |
| fungi | an alternative is to just drop the specific jobs which are no longer usable | 17:53 |
| *** dtantsur is now known as dtantsur|afk | 17:53 | |
| *** bgmccollum has quit IRC | 17:54 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstackid master: Disable legacy nodejs symlinks https://review.openstack.org/644956 | 17:58 |
| clarkb | fungi: ^ I think merging that then reinstalling nodejs on openstackid servers will fix the symlink issue | 17:58 |
| fungi | reinstalling is what i did previously, so hopefully this will make it stop undoing that? | 17:58 |
| fungi | and thanks! | 17:59 |
| fungi | i hadn't found time to look into it yet | 17:59 |
| fungi | also did you still want to take a peek at https://review.openstack.org/643476 to get wiki-dev01.openstack.org on track with xenial | 17:59 |
| fungi | ? | 17:59 |
| clarkb | as for why puppet 4 mattered there? I think it was just coincidence that the upstream pacakges must've updated at the same time | 17:59 |
| clarkb | fungi: I do! sorry yseterday was really weird scheduling wise. I'll look now | 18:00 |
| fungi | don't sweat it | 18:00 |
| mordred | clarkb, fungi: we could also investigate what would break if we bumped the stable/queens sdk constraint to 0.12. I'm PRETTY sure it would be a problem for senlin and heat and maybe masakari ... but I also think MAYBE we did the work in queens to allow them to support both calling syntaxes? | 18:00 |
| fungi | my whole week has been like that | 18:00 |
| mordred | like - sending in the bump patch should either fail spectacularly or be fine | 18:00 |
| mordred | and if it works, we could then go discuss suitability of such a thing in this case | 18:00 |
| fungi | mordred: i'm not opposed to the idea, but recommend bringing it up with prometheanfire and the requirements crwod | 18:00 |
| fungi | crowd | 18:01 |
| clarkb | mordred: ya though the requirements team tends to be pretty cautious about changes like that | 18:01 |
| clarkb | I guess as long as it is a constraints change and not a change in the acceptable range it is less scary | 18:01 |
| clarkb | fungi: virtual packages are magic | 18:01 |
| fungi | clarkb: virtual packages are a staple of debian and its derivatives... have been since the mid-90s | 18:02 |
| mordred | clarkb, fungi: yeah. I mean - I honestly have no idea if it'll work | 18:02 |
| *** armax has joined #openstack-infra | 18:02 | |
| clarkb | ya I know, but still magic. I am always happy for the build-essential meta package when I am on ap latform that doesn't udnerstand such things | 18:02 |
| tosky | fungi: about the lifecycle and the extended maintenance, just to be sure: the decision on whether running pike jobs or not for a certain project at this point is on the PTL shoulder? | 18:02 |
| fungi | tosky: it's up to the ptl to decide whether to have the core reviewers for the project continue maintenance, hand over maintenance to an extended maintenance team, or close down the branch for further development entirely | 18:04 |
| fungi | at least last i remember | 18:04 |
| *** priteau has quit IRC | 18:04 | |
| prometheanfire | sup? | 18:05 |
| prometheanfire | mordred: submit a proposed bump patch then in those projects (senlin/heat) submit a change that depends on the reqs change | 18:06 |
| *** armax has quit IRC | 18:06 | |
| mordred | prometheanfire: ok. it might totally crash and burn :) | 18:09 |
| fungi | we call that "openstack" ;) | 18:09 |
| prometheanfire | brokenstack | 18:09 |
| *** rpioso|afk is now known as rpioso | 18:11 | |
| clarkb | now to debug elastic-recheck | 18:11 |
| clarkb | there was a stale e-r process from march 12 holding the lock. Let me add a timeout of say 4 hours to those processes | 18:12 |
| mordred | prometheanfire: done. I listed the other patches in a comment on the patch | 18:14 |
| *** jpena is now known as jpena|off | 18:15 | |
| *** gfidente is now known as gfidente|afk | 18:15 | |
| prometheanfire | cool | 18:16 |
| *** bgmccollum has joined #openstack-infra | 18:17 | |
| *** e0ne has joined #openstack-infra | 18:17 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-elastic_recheck master: Timeout generation of e-r graphs https://review.openstack.org/644963 | 18:18 |
| clarkb | mriedem: ^ fyi that should improve this situation with e-r | 18:18 |
| clarkb | infra-root can I get reviwes on https://review.openstack.org/644963 and https://review.openstack.org/644956 to fixup a couple of recently noticed things | 18:19 |
| *** kopecmartin is now known as kopecmartin|off | 18:19 | |
| *** irclogbot_1 has quit IRC | 18:21 | |
| *** owalsh has quit IRC | 18:23 | |
| *** uberjay has quit IRC | 18:23 | |
| *** uberjay has joined #openstack-infra | 18:23 | |
| *** irclogbot_1 has joined #openstack-infra | 18:23 | |
| *** irclogbot_1 has quit IRC | 18:27 | |
| *** irclogbot_2 has joined #openstack-infra | 18:28 | |
| mriedem | clarkb: ack, yeah i noticed when i was running the graph locally yesterday it was pretty slow | 18:30 |
| mriedem | i've got a todo to cleanup old queries | 18:30 |
| mriedem | which i should really automate, but... | 18:30 |
| *** PrinzElvis has quit IRC | 18:30 | |
| *** evgenyl has quit IRC | 18:30 | |
| clarkb | actually I wonder if I want to flock timeout instead of timeout flock | 18:30 |
| clarkb | that way we don't kill the flock without cleaning up the process that writes things | 18:31 |
| *** icey has quit IRC | 18:31 | |
| clarkb | instead we kill the thing that writes things then let timeout/flock clean up the children | 18:31 |
| clarkb | ya /me is gonna flip it around | 18:31 |
| *** sweston has quit IRC | 18:31 | |
| *** dirk has quit IRC | 18:31 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-elastic_recheck master: Timeout generation of e-r graphs https://review.openstack.org/644963 | 18:32 |
| *** dougwig has quit IRC | 18:32 | |
| *** geekinutah has quit IRC | 18:32 | |
| *** evgenyl has joined #openstack-infra | 18:32 | |
| *** PrinzElvis_ has joined #openstack-infra | 18:33 | |
| *** icey has joined #openstack-infra | 18:33 | |
| *** sweston has joined #openstack-infra | 18:33 | |
| *** diablo_rojo has joined #openstack-infra | 18:33 | |
| *** geekinutah has joined #openstack-infra | 18:34 | |
| *** dirk has joined #openstack-infra | 18:34 | |
| *** dougwig has joined #openstack-infra | 18:45 | |
| corvus | dmsimard: +2 with suggested revision on 643394 | 18:47 |
| *** pgaxatte has quit IRC | 18:52 | |
| *** e0ne has quit IRC | 18:53 | |
| corvus | dmsimard: same on 643435 | 18:56 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to mirror a git repository to a remote git server https://review.openstack.org/643394 | 18:57 |
| frickler | corvus: for the ml post, maybe mention explicitly that ansible 2.5 will be EOL pretty soon? also, it's openstack-discuss now ;) | 18:59 |
| corvus | frickler: i think i did add that? and i sent it already | 18:59 |
| *** altlogbot_3 has quit IRC | 19:00 | |
| corvus | i think the bit about 2.5 being eol is in the first pgraph: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004034.html | 19:00 |
| *** altlogbot_0 has joined #openstack-infra | 19:01 | |
| frickler | ah, right, I had scrolled past that when I read the last part | 19:01 |
| frickler | corvus: still you did send to -dev , which may be confusing | 19:02 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add job and integration tests for the upload-git-mirror role https://review.openstack.org/643435 | 19:02 |
| dmsimard | corvus: thanks, addressed both | 19:02 |
| corvus | frickler: well, it got there. i can't unsend it now. if we don't want that sort of thing to happen, we should make -dev bounce. | 19:03 |
| dmsimard | corvus: also agreed that we might want to scope jobs file matchers better in general | 19:03 |
| frickler | we might also want to tackle the issue that the warnings which ansible 2.5 showed ended up in executor-debug.log only | 19:03 |
| frickler | if we can show them to users instead, they could fix them before they become errors with the next ansible version | 19:04 |
| frickler | if they never see them before it's too late, that's bad | 19:04 |
| frickler | fungi: do you think it would be time now to make the old MLs bounce or would you want to wait longer? | 19:05 |
| fungi | yeah, i think we're stuck just accepting openstack-dev messages for a while, since there are countless packages on pypi which list that as a maintainer address | 19:05 |
| corvus | frickler: i agree; mordred ^ could you help with that? | 19:05 |
| *** PrinzElvis_ is now known as PrinzElvis | 19:05 | |
| frickler | fungi: ah, that's a good point, agreed | 19:05 |
| *** altlogbot_0 has quit IRC | 19:06 | |
| fungi | even if we patch it in those hundreds of repos, that doesn't un-release all the older versions floating around which people might have and devide to follow the maintainer e-mail from | 19:06 |
| *** altlogbot_3 has joined #openstack-infra | 19:07 | |
| fungi | i expect there will come a time when we say, "okay, it's been long enough, let's shut down the old forwards," but that's probably years out by my reckoning | 19:07 |
| corvus | when we do, we can always make it bounce with a helpful message. | 19:07 |
| fungi | yep | 19:08 |
| fungi | i expect we would | 19:08 |
| *** lpetrut has joined #openstack-infra | 19:08 | |
| fungi | thankfully, exim makes that simple to do with an alias | 19:08 |
| fungi | openstack-dev: :fail: this is not the list you're looking for, try openstack-discuss! | 19:09 |
| corvus | clarkb, fungi, ianw, frickler: how does this email look for opendev scheduling? https://etherpad.openstack.org/p/Ip7tCf15vH | 19:10 |
| *** efried_rolling is now known as efried | 19:11 | |
| *** lpetrut has quit IRC | 19:12 | |
| fungi | corvus: looks great! the links check out fine too | 19:13 |
| *** irclogbot_2 has quit IRC | 19:16 | |
| frickler | corvus: fine for me, too | 19:16 |
| *** irclogbot_3 has joined #openstack-infra | 19:17 | |
| clarkb | I'll look after lunch | 19:20 |
| *** raissa has quit IRC | 19:26 | |
| *** irclogbot_3 has quit IRC | 19:30 | |
| *** quiquell has joined #openstack-infra | 19:31 | |
| *** irclogbot_1 has joined #openstack-infra | 19:31 | |
| *** quiquell has left #openstack-infra | 19:31 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid master: Disable legacy nodejs symlinks https://review.openstack.org/644956 | 19:33 |
| *** altlogbot_3 has quit IRC | 19:34 | |
| *** altlogbot_0 has joined #openstack-infra | 19:35 | |
| *** imacdonn has quit IRC | 19:40 | |
| *** imacdonn has joined #openstack-infra | 19:40 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki master: Switch from PHP5 to default PHP (PHP7 on Xenial) https://review.openstack.org/643476 | 19:42 |
| *** rascasoft has quit IRC | 19:48 | |
| *** eharney has quit IRC | 19:50 | |
| *** rascasoft has joined #openstack-infra | 19:50 | |
| clarkb | corvus: email looks good to me | 19:54 |
| *** dhellmann has quit IRC | 19:57 | |
| corvus | okay, sent to openstack, airship, starlingx, zuul | 19:58 |
| corvus | did i remember them all this time? :) | 19:58 |
| corvus | also... maybe we should start thinking about an opendev-announce list | 19:58 |
| clarkb | that would simplify the cross posting vs sending 5 emails problem | 19:59 |
| *** dhellmann has joined #openstack-infra | 19:59 | |
| corvus | fungi: are we ready to go ahead and merge https://review.openstack.org/642173 ? it should be okay to do so now, right? and then we just make dns changes later... | 20:02 |
| *** gfidente|afk has quit IRC | 20:02 | |
| *** eharney has joined #openstack-infra | 20:02 | |
| corvus | fungi: also, that implies coordinated dns changes with airshipit and starlingx, right? since they have elected not to use our system? | 20:02 |
| fungi | corvus: yes, i'd like to get 642173 merged sooner rather than later so i can test more easily | 20:04 |
| corvus | (presumably, if we hosted the redirects on the current git farm, we could make the change without dns changes) | 20:04 |
| fungi | ahh, that's what you mean by "elected not to use our system" (dns) | 20:04 |
| corvus | oh yeah, sorry | 20:05 |
| fungi | yeah we _could_ install these on the git servers but i thought a goal was to be able to turn those off | 20:05 |
| corvus | it is | 20:05 |
| *** armax has joined #openstack-infra | 20:06 | |
| fungi | anyway, i sort of figured we could test with local /etc/hosts edits and then just let the dns admins for those domains know when it's time to switch records | 20:06 |
| fungi | but i'm open to alternatives of course | 20:06 |
| corvus | right, but that's going to be friday on a holiday, so if that's the plan i guess we should start communicating it :) | 20:07 |
| clarkb | we have the ability to edit their dns for them | 20:08 |
| clarkb | we should totally communicate that we are going to do so if we do so | 20:09 |
| corvus | clarkb: oh are they all in the shared account? | 20:09 |
| openstackgerrit | melanie witt proposed openstack-infra/irc-meetings master: Update the nova team meeting chair to efried https://review.openstack.org/644987 | 20:09 |
| clarkb | corvus: yup | 20:09 |
| fungi | ahh, yeah i guess it's just tweaking in the rackspace dns service then | 20:10 |
| corvus | okay, i'm happy for us to do it then, that'll be easiest. i'll let the project contacts know | 20:10 |
| fungi | anyway just to set expectations i don't expect 642173 to actually cover those other domains at this point, but i wanted to get your initial git.o.o redirects copied in and tested before making additions | 20:11 |
| fungi | at least i assume it was only road-tested for git.o.o so far | 20:12 |
| corvus | fungi: oh, right, because we dropped the 'org' in some cases... | 20:12 |
| corvus | fungi: so yeah, we might need a slightly different version of that for each domain, where we implicitly add in the org | 20:12 |
| corvus | fungi: and yes, only tested with git.o.o | 20:13 |
| *** owalsh has joined #openstack-infra | 20:13 | |
| fungi | cool. i was going to take a stab at making one ruleset work for all of them, but if we have to do per-vhost versions it's not the end of the world. not like they're going to grow indefinitely | 20:15 |
| fungi | the alternative i considered was to key on lists of repository names | 20:16 |
| corvus | fungi: yeah, that could work. i haven't thought about which would be easier. | 20:16 |
| fungi | since in theory we only care about redirecting repositories which existed on those sites up to the point of the transition | 20:16 |
| fungi | if we do separate rules, we can probably do one for zuul and one for airship+starlingx, as the latter share a namespace | 20:17 |
| fungi | but with the list of "exceptional" repo names we could probably rewrite those to their git.openstack.org namespace counterparts first and then let them get matched against the subsequent rules? | 20:19 |
| openstackgerrit | Eric Fried proposed openstack-infra/irc-meetings master: nova-scheduler-meeting => placement-team-meeting https://review.openstack.org/644992 | 20:20 |
| clarkb | rewriting to their git.o.o name shouldn't be any different than writing to the premove gitea/opendev.org name right? | 20:20 |
| corvus | fungi: yes, but with all the cgit url variants, it's not clear to me that would be any easier than doing the final redirect to opendev. again -- i haven't really sat down and looked at it though. | 20:20 |
| fungi | sure. as i said, let's get this in place first and then it's a lot easier to experiment | 20:20 |
| clarkb | corvus: my ten second impression is that writing the final redirect is probably just as easy/hard and probably clearer to debug | 20:20 |
| corvus | fungi: i've upgraded my +2 to a +3 | 20:21 |
| fungi | thanks! | 20:21 |
| corvus | and i sent an email about dns | 20:21 |
| corvus | (er, a private email just to the two contacts of the affected projects) | 20:21 |
| fungi | appreciated! | 20:21 |
| *** eharney has quit IRC | 20:22 | |
| slaweq | clarkb: hi | 20:36 |
| clarkb | slaweq: hello | 20:36 |
| slaweq | clarkb: I have a question | 20:36 |
| clarkb | I hope I have answers :) | 20:36 |
| slaweq | clarkb: I'm debugging some multinode job, http://logs.openstack.org/84/643084/10/check/tobiko-devstack-neutron-fedora/f407fa3/job-output.txt.gz | 20:36 |
| slaweq | and it looks that devstack on both subnodes failed but job continues to run | 20:37 |
| slaweq | tests failed because of something else later but should it be like that? | 20:37 |
| slaweq | shouldn't job fail if devstack on one of nodes will fail? | 20:37 |
| *** pfallenop has joined #openstack-infra | 20:38 | |
| clarkb | slaweq: it looks like the job did fail https://review.openstack.org/#/c/643084/ is -1'd due to that job | 20:41 |
| clarkb | and ara records it as a failed playbook | 20:41 |
| clarkb | http://logs.openstack.org/84/643084/10/check/tobiko-devstack-neutron-fedora/f407fa3/ara-report/reports/7860b43d-5f74-444b-9218-5e14f79a6cf9.html | 20:41 |
| slaweq | clarkb: job failed because there was no nova-compute on which instance could be spawned | 20:41 |
| slaweq | but it didn't failed just after devstack on subnode-{1,2} failed | 20:42 |
| clarkb | I see so it doesn't fail until you run workload instead of failing at the devstack step | 20:42 |
| slaweq | but it continue to run test | 20:42 |
| slaweq | clarkb: exactly | 20:42 |
| slaweq | clarkb: is it "normal" and wanted behaviour? | 20:42 |
| slaweq | or bug? | 20:42 |
| clarkb | ansible's default is to stop running a playbook on the first task failure but you can override that. It is possible that this is a bug in the devstack role or perhaps even in ansible | 20:43 |
| clarkb | reading orchestrate-devstack role now | 20:43 |
| slaweq | I know that it isn't related only to this specific job | 20:44 |
| slaweq | I remember that I saw it some time ago in some neutron-tempest-plugin job | 20:44 |
| slaweq | but I then forgot to ask about it here :) | 20:45 |
| *** irclogbot_1 has quit IRC | 20:45 | |
| slaweq | and then it was even worst effect as job passed but in fact it was single node instead of expected multinode job because one node was not working properly | 20:45 |
| *** irclogbot_2 has joined #openstack-infra | 20:46 | |
| clarkb | slaweq: ok the reason is documented ansible behavior. https://docs.ansible.com/ansible/latest/user_guide/playbooks_error_handling.html#aborting-the-play By default it only stops running tasks on the host(s) that have failed tasks | 20:47 |
| clarkb | in this case your tasks fail on the subnodes but succeed on the primary. So it continues running on the primary until it fails. I believe that it will exit with failure regardelss | 20:47 |
| clarkb | its just doing more work than is necessary until it returns the failure state | 20:48 |
| clarkb | that means I don't think there is a correctness of result problem here, but there is a potential optomization if you want your jobs to return more quickly by setting any_errors_fatal | 20:48 |
| slaweq | clarkb: k, so I should configure "any_errors_fatal: true" in my job if I want to fail it fast, right? | 20:49 |
| corvus | https://docs.ansible.com/ansible/latest/user_guide/playbooks_delegation.html#interrupt-execution-on-any-error | 20:49 |
| corvus | it'd be in the playbook i think | 20:49 |
| slaweq | ok, clarkb and corvus thx for links and explanation | 20:50 |
| slaweq | I knew that You will know answer for my question :) | 20:50 |
| *** pfallenop has quit IRC | 20:51 | |
| *** pfallenop has joined #openstack-infra | 20:51 | |
| *** e0ne has joined #openstack-infra | 20:52 | |
| *** altlogbot_0 has quit IRC | 20:53 | |
| *** trown is now known as trown|outtypewww | 20:53 | |
| *** altlogbot_2 has joined #openstack-infra | 20:54 | |
| clarkb | fungi I'm going to manually reinstall the nodejs package on openstackid-dev now then double check pupept doesn't undo the symlinks | 21:00 |
| fungi | thanks! | 21:01 |
| fungi | also after that puppet-mediawiki patch i'm noticing a lot of other packages it's also not finding on xenial :/ | 21:02 |
| clarkb | that unbroke the loop. Now we wait ~10 minutes to see if it sticks | 21:02 |
| clarkb | I'll plan to merge https://review.openstack.org/#/c/643713/1/inventory/groups.yaml tomorrow and get more servers updated to puppet-4 with the openstackid issue likely behind us | 21:05 |
| *** kgiusti has left #openstack-infra | 21:05 | |
| clarkb | one of my favorite side effects of the bridge resize is ansible loop runs every 30 minutes now instead of every 45 :) | 21:06 |
| openstackgerrit | Merged openstack-infra/system-config master: Set up legacy git redirect sites https://review.openstack.org/642173 | 21:06 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck master: Timeout generation of e-r graphs https://review.openstack.org/644963 | 21:06 |
| fungi | yep, i was noticing that! | 21:06 |
| *** Lucas_Gray has joined #openstack-infra | 21:07 | |
| openstackgerrit | Malek Karray proposed openstack-infra/storyboard-webclient master: Flawed logic when checking for invalid query https://review.openstack.org/645008 | 21:09 |
| *** eharney has joined #openstack-infra | 21:19 | |
| clarkb | yup pupept is happy on openstackid-dev now | 21:19 |
| *** xek_ has quit IRC | 21:21 | |
| *** kjackal has quit IRC | 21:28 | |
| *** whoami-rajat has quit IRC | 21:29 | |
| corvus | i'm going to enable keep and verbose on all executors temporarily. | 21:37 |
| openstackgerrit | Merged openstack-dev/pbr master: Fix error when keywords are defined as a list in cfg https://review.openstack.org/639661 | 21:37 |
| corvus | okay, back to normal now | 21:41 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for nova functional test bug 1821092 https://review.openstack.org/645016 | 21:46 |
| openstack | bug 1821092 in OpenStack Compute (nova) "test_local_delete_removes_allocations_after_compute_restart failing since https://review.openstack.org/#/c/644591/" [Critical,Confirmed] https://launchpad.net/bugs/1821092 | 21:46 |
| ianw | this is probably getting a bit obscure, but is it intentional that the group vars in system-config/playbooks/zuul/templates don't seem to override those from playbooks/group_vars? | 21:57 |
| ianw | it seems they currently only contain fake secrets, which otherwise aren't defined in system-config | 21:57 |
| clarkb | ianw: I think everything under the zuul/ dir is meant for tests | 21:58 |
| corvus | ianw: yes, that is their purpose -- that replaces the secret hostvars on bridge in tests | 21:58 |
| ianw | corvus: right, but can we make them *override* values in system-config. for letsencrypt, i want to use a different global variable for CI (which just creates certs for test hosts) | 21:59 |
| *** e0ne has quit IRC | 22:00 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova functional test bug 1821092 https://review.openstack.org/645016 | 22:04 |
| openstack | bug 1821092 in OpenStack Compute (nova) "test_local_delete_removes_allocations_after_compute_restart failing since https://review.openstack.org/#/c/644591/" [Critical,Confirmed] https://launchpad.net/bugs/1821092 | 22:04 |
| corvus | ianw: the defining theory for the system-config tests is production simulation. so i think the current situation suggests that our config is set up so that we can not override a public value with a private one. i'm guessing that's something to do with the inventory file order. | 22:07 |
| corvus | ianw: have you tried setting the variable in zuul? | 22:07 |
| ianw | corvus: i think it's just "playbook values override inventory values" ... so when "ansible-playbook -v /home/zuul/src/git.openstack.org/openstack-infra/system-config/playbooks/base.yaml" runs, it's going to pick up ../group_vars in preference to the /etc/ ones | 22:09 |
| ianw | corvus: i have ... which lead me to https://review.openstack.org/#/c/644701/ ... i'm not sure how i feel about it yet | 22:09 |
| corvus | ianw: ah, yeah the vars in system-config/playbooks/zuul/templates get written to /etc/ansible/hosts/*_vars so that makes sense | 22:10 |
| ianw | i was just feeling like it would be better to not have in the production playbooks "if running under zuul generate certs for test01, test02, else <list of actual production certs>" if it could be helped | 22:11 |
| *** pcaruana has quit IRC | 22:11 | |
| ianw | i'll keep thinking ... it seems there's always another way with ansible ;) | 22:13 |
| corvus | ianw: it might be worth seeing if setting it as a zuul var overrides those, then maybe you can set it in the job | 22:14 |
| corvus | it's a little more verbose, but at least mantains separation of concerns | 22:15 |
| ianw | corvus: the issue with zuul vars has been that they don't make it into the *nested* ansible job (i.e. https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/zuul/run-base.yaml#n72) | 22:16 |
| corvus | ianw: oh of course, sorry | 22:16 |
| ianw | i was using 644701 to do that; set a flag in .zuul.yaml, which gets written in the inventory for the nested job | 22:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-elastic_recheck master: flock lockfile path comes before the command to lock https://review.openstack.org/645028 | 22:16 |
| clarkb | infra-root ^ fixes a derp that I totally derped. Reviews much appreciated | 22:17 |
| mordred | clarkb: lgtm. I'm not really here - but jetlag | 22:18 |
| clarkb | mordred: thank you | 22:22 |
| clarkb | mriedem: it is taking ~7 minutes to generate the gate pipeline results | 22:23 |
| clarkb | which is still quick enough that we hsouldn't start rolling every report into the next's 10 minutes to generate but slow enough that we do probably want to clean things up | 22:23 |
| *** mriedem is now known as mriedem_away | 22:25 | |
| mriedem_away | ack | 22:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 22:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 22:31 |
| clarkb | I'm sitting in the backyard because its the niceest winter day basically ever and my neighbors cat just killed a snake | 22:34 |
| fungi | clarkb: as of almost an hour ago it's now spring | 22:39 |
| fungi | at least in our hemisphere | 22:39 |
| *** arxcruz is now known as arxcruz|pto | 22:39 | |
| fungi | i guess you missed your opportunity to balance an egg on its end | 22:39 |
| clarkb | ah well yesterday was the nicest winter day then and today is the best first day of spring | 22:40 |
| *** yamamoto has joined #openstack-infra | 22:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix Ansible 2.7 uri module https://review.openstack.org/645034 | 22:46 |
| *** yamamoto has quit IRC | 22:49 | |
| *** Lucas_Gray has quit IRC | 22:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add missing Ansible invocation debug env variables https://review.openstack.org/645035 | 22:52 |
| *** armax has quit IRC | 22:55 | |
| *** rcernin has joined #openstack-infra | 22:57 | |
| *** rcernin has quit IRC | 22:58 | |
| *** rcernin has joined #openstack-infra | 22:58 | |
| dmsimard | btw I just landed what are probably the last big changes in the new ara api, it's pretty awesome imo but I'm biased | 22:59 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck master: flock lockfile path comes before the command to lock https://review.openstack.org/645028 | 23:01 |
| dmsimard | list of playbooks: https://api.demo.recordsansible.org/api/v1/playbooks -- or a playbook in detail: https://api.demo.recordsansible.org/api/v1/playbooks/1 | 23:01 |
| dmsimard | respectively represented in the web client: https://web.demo.recordsansible.org/playbooks and https://web.demo.recordsansible.org/playbooks/1 | 23:02 |
| dmsimard | There was not much progress on the web ui because the API wasn't ready but it's usable now | 23:03 |
| aspiers | clarkb: I was thinking in terms of time saved for humans inspecting massive log files for failures | 23:05 |
| clarkb | aspiers: ya but if someone looks at the file once caching it afte rthat doesn't help | 23:05 |
| aspiers | for example, myself earlier today :) | 23:05 |
| aspiers | no, it would have helped because I looked at it, then thought I was done so closed the tab, then realised I needed to look again, multiplied a few times | 23:06 |
| aspiers | sometimes that can span several days | 23:06 |
| clarkb | aspiers: hrm we've looked at access patterns before and the vast majority of files are only every fetched by our log processing pipeline and those that are opened more than that tend to get opened the once | 23:07 |
| aspiers | and while I realise keeping a million tabs open at all times is the most popular workflow, it doesn't work for me (not least because of RAM limitations) | 23:07 |
| clarkb | we can look at that again as things may have changed. | 23:07 |
| clarkb | fwiw I also encourage smaller log files for this reason and it makes it easie rt ofind the failure in the sea of text (in particular openstack is really bad at all the deprecation warnings flooding logs) | 23:08 |
| aspiers | yes totally agree but it's not realistically within my grasp to split up huge tempest runs into smaller files | 23:09 |
| aspiers | I appreciate I might be talking about a statistical corner case. My suggestion however is an optimisation for humans not for machines :) | 23:09 |
| clarkb | also not sure if it helps, but I often use vim to open the files then I can save them locally | 23:09 |
| clarkb | but I do that because firefox doesn't like 200MB log files and vim doesn't care | 23:10 |
| aspiers | In this case it wouldn't have helped, because on subsequent opens I needed to obtain a deeplink | 23:10 |
| aspiers | to share with another dev | 23:10 |
| *** rascasoft has quit IRC | 23:10 | |
| aspiers | But I only asked on the assumption that tweaking Apache to send a Last-Modified: header or similar would be trivial and have zero downsides. Maybe I was wrong on that? | 23:11 |
| clarkb | one thing we could probably do is set up the caching to only happen if the request comes from a webbrowser | 23:11 |
| clarkb | and assume the other clients are either bots or my vim with local storage | 23:11 |
| aspiers | I mean, this is client-side caching we are talking about | 23:11 |
| aspiers | At least, that's what I meant | 23:11 |
| clarkb | oh yo uwant client side caching. I read it as server side to improve the render time | 23:11 |
| aspiers | Noooooo | 23:11 |
| aspiers | Just so the browser knows that it doesn't need to redownload 20MB of text | 23:11 |
| clarkb | ya we could set generous ttls in headers since those files aren't expected to get modified once written | 23:12 |
| aspiers | Exactly what I meant | 23:12 |
| fungi | but also web browsers don't generally like to cache files above a certain size | 23:12 |
| aspiers | Oh? | 23:14 |
| aspiers | Anyway, it's just an idea - take it or leave it ;-) | 23:14 |
| aspiers | It wouldn't be the first dumb idea I've had ;-) | 23:14 |
| clarkb | fungi: I think most of our files are small enough to get cached | 23:14 |
| fungi | setting a long ttl on those seems fine to me since they never change until they get deleted | 23:14 |
| clarkb | fungi: and its only the browser killers you'll be using vim with anyway tha won't | 23:14 |
| clarkb | set https://httpd.apache.org/docs/current/mod/mod_expires.html expiresbytype for text/html to access plus 6 weeks? | 23:22 |
| *** jamesmcarthur has quit IRC | 23:23 | |
| clarkb | that should work reasonably well | 23:23 |
| *** mriedem_away has quit IRC | 23:28 | |
| *** mriedem has joined #openstack-infra | 23:29 | |
| *** tosky has quit IRC | 23:31 | |
| *** jamesmcarthur has joined #openstack-infra | 23:32 | |
| *** jamesmcarthur has quit IRC | 23:38 | |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| *** rf0lc0 has joined #openstack-infra | 23:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] letsencrypt support https://review.openstack.org/636759 | 23:47 |
| *** rfolco has quit IRC | 23:49 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove old queries for stale and/or fixed bugs https://review.openstack.org/645040 | 23:50 |
| mriedem | clarkb: there you go ^ | 23:50 |
| *** rascasoft has joined #openstack-infra | 23:51 | |
| mriedem | pabelanger: do you know if we can close this out? http://status.openstack.org/elastic-recheck/#1709744 | 23:51 |
| pabelanger | mriedem: yah, I think so. we have a reverse proxy in place for them now | 23:52 |
| *** sthussey has quit IRC | 23:53 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove old queries for stale and/or fixed bugs https://review.openstack.org/645040 | 23:53 |
| mriedem | ack thanks | 23:54 |
| *** rascasoft has quit IRC | 23:57 | |
| *** rlandy has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!