| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 00:02 |
|---|---|---|
| openstackgerrit | Merged zuul/zuul-jobs master: Add multi-distro support to install-docker https://review.opendev.org/664455 | 00:11 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 00:11 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 00:12 |
| *** michael-beaver has quit IRC | 00:17 | |
| *** ianychoi_ has joined #zuul | 00:18 | |
| *** ianychoi has quit IRC | 00:20 | |
| *** jamesmcarthur has quit IRC | 00:39 | |
| *** mhu has quit IRC | 00:40 | |
| *** pabelanger has quit IRC | 00:40 | |
| *** jpena|off has quit IRC | 00:40 | |
| *** fbo has quit IRC | 00:41 | |
| *** mhu has joined #zuul | 00:45 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 00:46 |
| *** jpena|off has joined #zuul | 00:48 | |
| *** igordc has quit IRC | 01:28 | |
| *** spsurya has joined #zuul | 01:51 | |
| *** pabelanger has joined #zuul | 02:25 | |
| *** bhavikdbavishi has joined #zuul | 02:49 | |
| *** kklimonda has quit IRC | 03:48 | |
| *** kklimonda has joined #zuul | 03:48 | |
| *** evgenyl has quit IRC | 03:49 | |
| *** evgenyl has joined #zuul | 03:49 | |
| *** fdegir has quit IRC | 04:08 | |
| *** fdegir has joined #zuul | 04:09 | |
| *** bhavikdbavishi has quit IRC | 04:31 | |
| *** bhavikdbavishi has joined #zuul | 04:33 | |
| *** pcaruana has joined #zuul | 04:54 | |
| *** pcaruana has quit IRC | 04:59 | |
| *** raukadah is now known as chandankumar | 05:01 | |
| *** saneax has joined #zuul | 05:07 | |
| *** saneax has quit IRC | 05:15 | |
| *** igordc has joined #zuul | 05:58 | |
| *** gtema has joined #zuul | 06:31 | |
| *** [GNU] has left #zuul | 06:34 | |
| *** gtema_ has joined #zuul | 06:35 | |
| *** gtema has quit IRC | 06:38 | |
| *** pcaruana has joined #zuul | 06:56 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:13 |
| *** dmellado has quit IRC | 07:17 | |
| *** saneax has joined #zuul | 07:19 | |
| *** gtema_ is now known as gtema | 07:21 | |
| *** fbo has joined #zuul | 07:25 | |
| *** bhavikdbavishi has quit IRC | 07:26 | |
| *** hashar has joined #zuul | 07:28 | |
| *** dmellado has joined #zuul | 07:29 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 07:37 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] Docker compose example: add keycloak authentication https://review.opendev.org/664813 | 07:40 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:41 |
| *** hashar has quit IRC | 07:49 | |
| *** jangutter_ has joined #zuul | 07:53 | |
| *** jangutter has quit IRC | 07:56 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** hashar has joined #zuul | 07:59 | |
| *** hashar has quit IRC | 08:07 | |
| *** hashar has joined #zuul | 08:09 | |
| *** igordc has quit IRC | 08:17 | |
| *** hashar has quit IRC | 08:18 | |
| *** hashar has joined #zuul | 08:19 | |
| *** hashar_ has joined #zuul | 08:24 | |
| *** hashar has quit IRC | 08:24 | |
| *** hashar_ has quit IRC | 08:25 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 08:32 |
| *** bhavikdbavishi has joined #zuul | 08:49 | |
| *** hashar has joined #zuul | 08:58 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 08:59 |
| *** hashar has quit IRC | 09:31 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:44 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 09:55 |
| pabelanger | corvus: tobiash: ^ is a basic attempt to add retry logic for github driver and event processing, would love feedback | 09:55 |
| *** jangutter has joined #zuul | 10:02 | |
| *** jangutter_ has quit IRC | 10:06 | |
| *** bhavikdbavishi has quit IRC | 10:18 | |
| *** gtema has quit IRC | 10:27 | |
| *** gtema has joined #zuul | 10:27 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 10:27 |
| pabelanger | I also wonder if we should maybe step into each event handler and wrap retry logic closer to api functions for github? eg: getPullBySha() | 10:30 |
| pabelanger | but like I said, feedback welcome :) | 10:30 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:12 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:17 |
| *** electrofelix has joined #zuul | 11:28 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** bhavikdbavishi has joined #zuul | 11:44 | |
| *** gtema has quit IRC | 11:45 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:50 |
| ofosos | How do you guys run a single test in the current zuul test suite? Like the `tox ...' invocation. | 11:51 |
| *** rlandy has joined #zuul | 11:55 | |
| *** rlandy is now known as rlandy|ruck | 12:05 | |
| *** sanjayu_ has joined #zuul | 12:05 | |
| *** saneax has quit IRC | 12:08 | |
| *** gtema has joined #zuul | 12:09 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 12:39 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 12:41 |
| *** jamesmcarthur has joined #zuul | 12:41 | |
| *** michael-beaver has joined #zuul | 12:42 | |
| *** bhavikdbavishi has quit IRC | 12:50 | |
| tobiash | pabelanger: I'll look later today | 12:51 |
| *** bhavikdbavishi has joined #zuul | 12:51 | |
| *** bhavikdbavishi has quit IRC | 12:52 | |
| *** zbr|rover is now known as zbr|flow | 12:53 | |
| *** bhavikdbavishi has joined #zuul | 12:54 | |
| SpamapS | ofosos: just pass the import path of the test. so tox -epy36 tests.unit.test_git_driver.TestGitDriver.test_basic | 13:01 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 13:02 |
| SpamapS | ofosos:also note, there's a useful little pip installable utility, ttrun, which I use a lot to run individual tests so I can be attached directly to the tests' stdout/stderr. | 13:03 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:04 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| ofosos | SpamapS: thanks, ttrun works really well :) | 13:19 |
| Shrews | ofosos: tox and ttrun will use regex on the supplied test name, so it may still run multiple tests if they match. Running stestr directly can select a single test without the regex | 13:29 |
| *** pcaruana has quit IRC | 13:30 | |
| *** pcaruana|afk| has joined #zuul | 13:30 | |
| Shrews | something like: source .tox/py56/bin/activate ; stestr run -n path.to.test | 13:30 |
| *** bhavikdbavishi has quit IRC | 13:32 | |
| *** electrofelix has quit IRC | 13:37 | |
| openstackgerrit | Joshua Hesketh proposed zuul/zuul master: WIP Expose date time as facts https://review.opendev.org/664674 | 13:38 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 13:39 | |
| *** jamesmcarthur has joined #zuul | 13:46 | |
| pabelanger | tobiash: great, thanks | 13:56 |
| mhu | Hello, any blocker on this one? https://review.opendev.org/#/c/535541/ It'd be good to see it merged so I can document the protected REST API in the zuul_admin_web patch chain properly | 14:11 |
| *** jhesketh has joined #zuul | 14:18 | |
| *** swest has quit IRC | 14:19 | |
| *** ianychoi_ is now known as ianychoi | 14:22 | |
| clarkb | pabelanger: on phone and that fails at inline gerrit comments, but it might be good to catch the github server error exception instead of all exceptions if we aregoing to retry | 14:23 |
| clarkb | this way more fatal errors dont force us into a 5 second pause per event? | 14:23 |
| pabelanger | clarkb: yah, was thinking that too | 14:23 |
| *** igordc has joined #zuul | 14:26 | |
| corvus | mhu: there are errors in the yarn.lock file in that patch; i'm very surprised that the js jobs worked... | 14:37 |
| mhu | corvus, oh my bad, I basically hacked and slashed into that file hoping it'd work as a rebase | 14:38 |
| mhu | what's the right way to regen the lock when doing a rebase? | 14:39 |
| corvus | mhu: maybe just checkout the current yarn.lock from master then re-run the add command? | 14:39 |
| tobiash | mhu: yarn install for additional deps, and yarn update for updating all deps | 14:40 |
| tobiash | the yarn.lock file should never be modified manually | 14:40 |
| corvus | tobiash: right, but during a rebase it is modified "manually" (in that git does the modification). | 14:41 |
| tobiash | oh, I guess it should be regenerated after a rebase | 14:41 |
| clarkb | I think you rebase, rm the file, regen the file, git commit --amend | 14:41 |
| clarkb | rm step may not be necessary | 14:42 |
| corvus | you'll need a 'yarn install' in there too to "re-add" the new package the patch originally added | 14:43 |
| pabelanger | clarkb: should we still trap all exceptions, just not retry, right? | 14:43 |
| corvus | oh, sorry, you won't need to re-add because it's in package.json | 14:44 |
| clarkb | pabelanger: ya | 14:44 |
| corvus | so yeah, what tobiash and clarkb said :) | 14:44 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 14:49 |
| pabelanger | clarkb: okay, I think ^ does what you ask | 14:50 |
| *** rfolco has quit IRC | 14:52 | |
| SpamapS | Are the rendered rst's from zuul-jobs published somewhere? | 14:52 |
| pabelanger | SpamapS: https://zuul-ci.org/docs/zuul-jobs/ | 14:53 |
| fungi | findable from https://zuul-ci.org/docs/ too | 14:53 |
| SpamapS | oh, duh | 14:54 |
| SpamapS | I looked right past that in the docs page | 14:54 |
| fungi | or via the documentation drop-down | 14:54 |
| fungi | in the site's top navbar | 14:54 |
| fungi | we're well aware that website could use some work to be more readable, so if you think of a better way to organize that stuff it's all in the zuul-website repo in gerrit | 14:55 |
| fungi | it's certainly far from perfect | 14:56 |
| openstackgerrit | Merged zuul/zuul master: Fix minor typos in Nodepool OpenStack instructions https://review.opendev.org/664665 | 14:57 |
| SpamapS | I think it's pretty good, but what's missing, glaringly, are examples. | 14:57 |
| *** sanjayu_ has quit IRC | 14:58 | |
| fungi | that would be swell, agreed | 14:58 |
| SpamapS | I'm holding hands with a new user who is adventurous enough to try and write their own jobs... we'll see how it goes. | 14:58 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Unify gearman worker handling https://review.opendev.org/664948 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Move fingergw config to fingergw https://review.opendev.org/664949 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Support ssl encrypted fingergw clients https://review.opendev.org/664950 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Add ssl support to fingergw client https://review.opendev.org/664951 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Register finger gateway if it's zoned https://review.opendev.org/664952 | 15:00 |
| fungi | we have the quick-start tutorial but it stops short of providing example jobs beyond the base job | 15:00 |
| SpamapS | Yeah I think we might want to just start dropping zuul config snippets in the job docs, and ansible playbook snippets in the role docs | 15:01 |
| SpamapS | also I'm surprised how few roles we have in there | 15:01 |
| fungi | if you skim through https://zuul-ci.org/docs/zuul/user/config.html#job you'll see a handful of examples at least | 15:03 |
| fungi | assuming what you're looking for are job examples and not role or playbook examples | 15:04 |
| fungi | i mostly expected folks would browse through what's in the zuul-jobs repo for examples, but those are of course less useful as examples of jobs which use the zuul-jobs repo as a source of building blocks | 15:05 |
| SpamapS | https://www.terraform.io/docs/providers/openstack/r/compute_instance_v2.html | 15:07 |
| SpamapS | I want it to be as simple as that ^^ | 15:07 |
| SpamapS | Like "here's how it looks when you use it" | 15:07 |
| SpamapS | The config.html thing is definitely good, but that's more "here's how zuul config generally works" | 15:08 |
| * SpamapS will try and add a few examples to demonstrate | 15:08 | |
| fungi | awesome, thanks! | 15:09 |
| fungi | i've got tests.unit.test_v3.TestCentralJobs (nondeterministically?) hitting what looks like an unclean shutdown in tox-py36... can anybody interpret better? http://logs.openstack.org/70/662870/8/gate/tox-py36/4b27eb7/testr_results.html.gz | 15:09 |
| fungi | ran into this on the gate pass of a change which was rechecked and made it through check just fine, so it certainly seems to be an inconsistent result at any rate | 15:10 |
| Shrews | fungi: odd. i'm seeing some odd failures in the plugin unit tests too. e.x.) http://logs.openstack.org/62/663762/10/check/tox-py36/e550cd7/job-output.txt.gz#_2019-06-12_13_49_54_042249 | 15:15 |
| Shrews | also http://logs.openstack.org/62/663762/10/check/tox-py35/db3efff/job-output.txt.gz#_2019-06-12_13_41_33_637182 | 15:17 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 15:17 | |
| Shrews | note the change that depends on that one passed | 15:17 |
| fungi | tests.unit.test_v3.TestAnsible28.test_plugins [260.657389s] ... FAILED | 15:19 |
| fungi | fixtures._fixtures.timeout.TimeoutException | 15:19 |
| fungi | we set wait_timeout=180 in test_plugins() | 15:20 |
| fungi | but i'm guessing this is not what actually sets the test timeout? | 15:20 |
| fungi | since the test was allowed to run for 260s before getting terminated | 15:21 |
| fungi | i'm also seeing it on 662870 where i raise wait_timeout to 270 and it still dies at ~260s | 15:21 |
| clarkb | the test timeout is set in the base job as a fixture | 15:22 |
| fungi | i think the default (testr?) test timeout is 120s so we must be setting it higher somewhere | 15:22 |
| clarkb | er baase test | 15:22 |
| fungi | ahh, i'll check there | 15:22 |
| fungi | thanks | 15:22 |
| fungi | is that the wait_timeout in BaseTestCase() or something else? | 15:25 |
| fungi | ahh, in setUp() we have os.environ.get('OS_TEST_TIMEOUT', 0) which is passed to fixtures.Timeout() | 15:26 |
| fungi | looks like we hard kill the test 20 seconds after that | 15:27 |
| fungi | so we must be defaulting to 240s for that test_timeout value | 15:27 |
| fungi | coming from tox.ini which sets OS_TEST_TIMEOUT=240 | 15:28 |
| fungi | so the test dying at ~260s makes sense at least | 15:28 |
| fungi | but it's unclear to me how we'd override that for a specific test while still respecting the envvar and soft/hard logic | 15:29 |
| *** pcaruana|afk| has quit IRC | 15:30 | |
| fungi | i don't see any examples in the existing corpus of tests where we override that | 15:30 |
| clarkb | I think we generally bump the global timeout | 15:32 |
| clarkb | its there to prevent the unittests running indefinitely so a few extra minutes is probably fine | 15:32 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 15:33 |
| *** hashar has joined #zuul | 15:37 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Safely add Ansible password lookup plugin https://review.opendev.org/662870 | 15:42 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Increase test timeout to 6 minutes https://review.opendev.org/664961 | 15:42 |
| fungi | clarkb: Shrews: 664961 will hopefully solve that | 15:42 |
| clarkb | another more involved approach is to split tests into smaller chunks | 15:46 |
| fungi | right | 15:48 |
| fungi | corvus suggested we could shard the plugins testing | 15:48 |
| fungi | though not too heavily as the setup costs for them are significant | 15:49 |
| tobiash | fungi: I think that makes sense, that's one of the most expensive tests | 15:49 |
| clarkb | pabelanger: commented on the retry loop change | 15:50 |
| *** jangutter has quit IRC | 15:51 | |
| pabelanger | clarkb: ack | 15:53 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Register finger gateway if it's zoned https://review.opendev.org/664952 | 15:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Route streams to different zones via finger gateway https://review.opendev.org/664965 | 15:54 |
| pabelanger | clarkb: do you want to trap 'github3.exceptions.GitHubException' outside of the catch all exception or just change it to more specific expection? | 15:56 |
| clarkb | pabelanger: I only want to retry if the error is related to github specific exceptions that we expect will succeed on a retry | 15:58 |
| clarkb | and not rely on every other exception having a break to do that. Though rereading it with comments from a previous ps that is due to the else? | 15:59 |
| clarkb | (mostly i'm worried if we add handling for more exceptions we'll end up stuck in the loop longer than expected) | 15:59 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 15:59 |
| pabelanger | clarkb: okay, ^ just retries the current expection I see, 502. However, there maybe more, but haven't seen it yet. | 16:00 |
| pabelanger | the for / else was mostly for logging that we couldn't get the info, and maybe in the future report back to PR of the failure | 16:00 |
| clarkb | pabelanger: I think we can drop the else due to the general exception catching | 16:01 |
| *** gtema has quit IRC | 16:01 | |
| pabelanger | kk | 16:01 |
| clarkb | and ya the general exception can go up a level | 16:01 |
| pabelanger | clarkb: up a level? | 16:02 |
| clarkb | ya writing up a comment | 16:02 |
| *** hashar has quit IRC | 16:02 | |
| pabelanger | thanks | 16:03 |
| clarkb | done | 16:05 |
| clarkb | er it needs the one break | 16:06 |
| pabelanger | clarkb: I see now, thanks | 16:08 |
| *** pcaruana|afk| has joined #zuul | 16:09 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Bump default_read_timeout for github driver to 300 https://review.opendev.org/664667 | 16:10 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 16:10 |
| pabelanger | k, I also rebased away WIP on parent | 16:10 |
| clarkb | pabelanger: ya thats it, and its 11 lines shorter diff :) | 16:11 |
| pabelanger | +1 | 16:11 |
| *** bhavikdbavishi has joined #zuul | 16:12 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 16:12 | |
| *** mattw4 has joined #zuul | 16:13 | |
| pabelanger | I also updated the ML post with latest info | 16:23 |
| *** bhavikdbavishi1 has joined #zuul | 16:31 | |
| *** bhavikdbavishi has quit IRC | 16:32 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:32 | |
| Shrews | fungi: thx for the timeout patch. these timeout failures are getting annoyingly consistent for some reason | 16:36 |
| fungi | Shrews: likely we're getting some slower nodes with greater frequency than usual in opendev | 16:42 |
| smcginnis | clouds.yaml and container question for folks - I | 16:44 |
| smcginnis | I'm using https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples as a base to build off of. | 16:44 |
| corvus | tobiash: can you +3 https://review.opendev.org/661316 ? | 16:45 |
| smcginnis | I've updated my nodepool config to use my openstack cloud. I'm including a cloud.yaml that gets mounted in /etc/openstack in the container. | 16:45 |
| corvus | (clouds.yaml with an s, but i'm assuming you got that) | 16:45 |
| smcginnis | I'm not seeing any activity though. Should I see something in the executor logs of at least trying to connect to my cloud? | 16:45 |
| smcginnis | corvus: Yes, sorry, I meant clouds.yaml. | 16:46 |
| corvus | smcginnis: zuul doesn't know anything about openstack at all -- that's nodepool, so you'd look in the nodepool launcher container for logs | 16:46 |
| corvus | smcginnis: this container: https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples/docker-compose.yaml#L94 | 16:46 |
| smcginnis | Ah, that would be "example_node_1", right? | 16:46 |
| corvus | smcginnis: example_node_1 is a static worker node. in the quickstart, nodepool is only configured to use the static node driver, with that node as the static node. | 16:47 |
| corvus | smcginnis: so you will also need to change the nodepool config to tell it to use openstack | 16:47 |
| corvus | smcginnis: here's the nodepool config which gets bind-mounted into the nodepool launcher container: https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples/etc_nodepool/nodepool.yaml | 16:48 |
| smcginnis | corvus: Yep, updated etc_nodepool/nodepool.yaml with that and added my nodepool key to etc_zuul/zuul.yaml. | 16:48 |
| tobiash | corvus: done | 16:49 |
| corvus | smcginnis: ah, ok. the next things i would do would be to drop the static driver from that config if you didn't already (just to make sure that everything nodepool does from there out is with openstack). you can add the "min-ready" option to your openstack nodepool config to tell nodepool to spin up nodes before zuul requests them. that will help you get everything working with nodepool without having to | 16:49 |
| corvus | get zuul involved. | 16:49 |
| corvus | smcginnis: basically, setting "min-ready: 1" will tell nodepool to always make sure there is a node ready. then you can check the launcher logs to see how it's doing spinning up that node. | 16:50 |
| corvus | smcginnis: https://zuul-ci.org/docs/nodepool/configuration.html#attr-labels.min-ready | 16:50 |
| smcginnis | I set min-ready to 1, but I never see any activity. Shouldn't there be an "example_launcher_1" container created? | 16:50 |
| Shrews | smcginnis: if you use the openstack driver, no containers are used. the example_launcher_1 container is used with the static driver | 16:51 |
| smcginnis | Shrews: Oh, I assumed the launcher was needed if you were *not* using static. | 16:52 |
| Shrews | smcginnis: umm, i think wires are getting crossed here :) | 16:52 |
| smcginnis | I'm still learning, so probably. :) I had just assumed nodepool-launcher was what would use the openstack driver to launch new instances. | 16:53 |
| corvus | smcginnis: docker-compose-up should have created a container called "example_launcher_1" when you first ran it in which the nodepool launcher is run. without that, zuul would be unable to run any jobs which require nodes at all. | 16:53 |
| Shrews | corvus: what are the static nodes named? 'node'? | 16:54 |
| smcginnis | I did have an example_node_1 container. | 16:54 |
| corvus | smcginnis: it does that, but it uses whatever driver you tell it, so it also manages static instances | 16:54 |
| corvus | Shrews: yes | 16:54 |
| Shrews | ah yeah. ok | 16:54 |
| smcginnis | No launcher though, which I guess sounds right then since I've since switched to the openstack driver. | 16:54 |
| corvus | it's just one static node atm | 16:54 |
| corvus | smcginnis: you should find out why your example_launcher_1 container is not running. perhaps it crashed, and docker logs will tell you why | 16:55 |
| Shrews | smcginnis: i probably confused the discussion then. it's the 'node' container that you don't need | 16:55 |
| smcginnis | Shrews: OK, that makes a little more sense. | 16:55 |
| smcginnis | And since re-docker-compose-up'ing I do now see a launcher_1. | 16:56 |
| smcginnis | No node created on my cloud, but at least I have somewhere to look for clues now. Thanks! | 16:56 |
| smcginnis | And bingo, looks like it doesn't like my clouds.yaml: "keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect." | 16:57 |
| tobiash | pabelanger: retry lgtm | 16:58 |
| tobiash | pabelanger: but the note probably will never be addressed because we don't know in advance if we would trigger something so we cannot decide if to report | 16:59 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 17:01 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 17:01 |
| *** rfolco has joined #zuul | 17:02 | |
| smcginnis | OK, next question (probably an easy one that I don't really want to hear the answer to) - can things work with provider networking, or does nodepool assume the "typical" Neutron networking? | 17:03 |
| smcginnis | http://paste.openstack.org/show/752828/ | 17:03 |
| tobiash | smcginnis: sure it just works normally | 17:04 |
| tobiash | You might need to mark the network as routes_externally in clouds.yaml | 17:05 |
| pabelanger | tobiash: thanks for review, I also agree | 17:05 |
| *** mattw4 has quit IRC | 17:05 | |
| smcginnis | tobiash: Thanks, I'll give that a shot. | 17:05 |
| tobiash | smcginnis: If the network is marked like this nodepool won't try to assign a fip | 17:06 |
| fungi | you can also explicitly indicate it in the network info in your clouds.yaml | 17:06 |
| fungi | (see the openstacksdk docs for details) | 17:06 |
| smcginnis | So regions.values.networks.routes_externally: true? | 17:07 |
| fungi | nodepool basically just asks the sdk to figure out an externally-routable interface, so whatever it takes to convince the sdk of that | 17:07 |
| *** mattw4 has joined #zuul | 17:07 | |
| tobiash | smcginnis: like this: http://paste.openstack.org/show/752829/ | 17:08 |
| fungi | https://docs.openstack.org/openstacksdk/latest/user/config/network-config.html | 17:09 |
| fungi | looks like yes | 17:09 |
| fungi | the openstack driver docs for nodepool lead there, but it's sorta buried here https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].cloud | 17:11 |
| smcginnis | tobiash: "networks" or "regions"? | 17:11 |
| tobiash | I have networks | 17:11 |
| tobiash | I have one cloud with provider network too | 17:12 |
| *** jamesmcarthur has quit IRC | 17:12 | |
| fungi | the sdk example also does it in your networks list | 17:12 |
| *** jpena is now known as jpena|off | 17:13 | |
| smcginnis | OK, the only thing I came across before was https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html#per-region-settings | 17:13 |
| smcginnis | Using networks, but I'm still getting the "Neutron returned NotFound for floating IPs" message. | 17:14 |
| smcginnis | Must have typo'd something. | 17:14 |
| tobiash | smcginnis: if you use regions, then networks within the region | 17:14 |
| tobiash | I don't have regions there | 17:14 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 17:14 |
| fungi | you probably also have to restart the launcher container if you change the nodepool config | 17:14 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 17:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test base-test https://review.opendev.org/664981 | 17:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test base-test https://review.opendev.org/664981 | 17:16 |
| smcginnis | tobiash, fungi: Anything obvious with this: http://paste.openstack.org/show/752830/ | 17:17 |
| tobiash | that looks correct according to the docs above, is the network name correct? | 17:18 |
| tobiash | and does nodepool use the same network? | 17:18 |
| smcginnis | Yep, double checked and my network is "provider". They are all on the same subnet. | 17:19 |
| tobiash | is this the only network visible in the tenant? | 17:19 |
| *** spsurya has quit IRC | 17:19 | |
| smcginnis | Yep, only network defined at all. | 17:19 |
| smcginnis | http://paste.openstack.org/show/752831/ | 17:20 |
| tobiash | can you try without specifying the regions in the clouds.yaml? | 17:21 |
| tobiash | (like in my example) | 17:21 |
| smcginnis | Coming right up. | 17:21 |
| pabelanger | https://github.com/ansible-network/windmill-config/blob/master/nodepool/clouds.yaml.j2#L9 is an example of using regions, we do that on limestone to fix an issue with on of their networks. Mind you, it might have been fixed in openstasksdk now too | 17:21 |
| tobiash | that works with recent nodepool and my ancient juno or icehouse based cloud | 17:22 |
| fungi | https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack] also has a sizeable providers example if you're looking for one to compare against | 17:23 |
| smcginnis | Same error. Would I maybe also need floating_ip_source: None | 17:23 |
| tobiash | the floatingip-source way is also worth a try | 17:23 |
| *** ianychoi has quit IRC | 17:25 | |
| smcginnis | OK, adding that got rid of the error message. | 17:25 |
| tobiash | :) | 17:25 |
| smcginnis | Still no instance created. Should that be fairly quickly after starting, or does it take some time to realize it needs to create one? | 17:26 |
| fungi | out of curiosity, does your pools list entry in the nodepool config have a networks list containing that network name? | 17:26 |
| smcginnis | fungi: No, it does not. | 17:26 |
| smcginnis | http://paste.openstack.org/show/752832/ | 17:27 |
| fungi | the examples i linked in the driver documentation there have it, but not sure if it would have helped | 17:27 |
| fungi | presumably if there is only one network available, specifying it in the nodepool config should be unnecessary | 17:27 |
| fungi | is nodepool configured to use an existing image in your cloud, or to build and upload one? if the latter, the nodepool-builder will activate to create the image and then upload it to the provider | 17:28 |
| smcginnis | If I have http://paste.openstack.org/show/752832/ right, it should be using my existing image "Ubuntu-18.04" | 17:28 |
| smcginnis | At least that was my intention. | 17:29 |
| fungi | ahh, i see in the pool config you have an ubuntu-bionic label using the Ubuntu-18.04 image | 17:29 |
| fungi | yeah | 17:29 |
| corvus | let me update that config fore you, 1 sec | 17:29 |
| corvus | smcginnis: i think it should look more like this: https://etherpad.openstack.org/p/tPSbGVxjl6 | 17:31 |
| pabelanger | +1 | 17:31 |
| smcginnis | corvus: Oh, thanks. I'll try that. | 17:31 |
| smcginnis | No errors, but still no instances being created. | 17:34 |
| tobiash | smcginnis: what does 'nodepool list' say? | 17:34 |
| smcginnis | tobiash: I put the output at the bottom of https://etherpad.openstack.org/p/tPSbGVxjl6 | 17:36 |
| pabelanger | you likely need another name then ubuntu-bionic, since that might be the min-ready of 1 | 17:36 |
| pabelanger | for the label | 17:36 |
| tobiash | smcginnis: there is a ready node listed, that's why nodepool doesn't create another ine | 17:36 |
| smcginnis | Is that left over from the static node? "static-vms" | 17:37 |
| tobiash | oh yes, can you try 'nodepool delete 0000000003' ? | 17:37 |
| pabelanger | smcginnis: https://zuul-ci.org/docs/nodepool/operation.html#removing-a-provider is likely the step you missed | 17:38 |
| tobiash | if that doesn't work, you need to delete that znode from zookeeper | 17:38 |
| corvus | it may not know how to delete it with the static driver gone, so you may need to put the config for that back and set max-servers to 0 and remove it | 17:38 |
| corvus | don't delete the node from zk | 17:38 |
| pabelanger | agree with corvus | 17:38 |
| smcginnis | pabelanger: Ah, thanks. I should have read more. I just stopped containers, updated the configs, and started again. | 17:38 |
| corvus | smcginnis: or, if you want, you could delete the zk container and start over :) | 17:39 |
| smcginnis | \o/ ubuntu-bionic-mycloud-0000000004 | 17:39 |
| pabelanger | yay | 17:40 |
| smcginnis | Looks like that did it. I will try to get these steps written down to make sure it creates a clean config. | 17:40 |
| tobiash | pabelanger: if you have time, this would be a small review that cleans up leftover from multi-ansible: https://review.opendev.org/661435 | 17:40 |
| smcginnis | Thanks again all. Still learning, but hopefully once I get this all written down it will prevent future questions being repetitive. | 17:40 |
| fungi | figuring out why you needed to explicitly disable fip in the config would probably also be a good idea | 17:41 |
| smcginnis | Maybe the log message was just a red herring? | 17:41 |
| pabelanger | tobiash: sure, will look shortly, going to take a walk down to lake. Is real nice out today :D | 17:41 |
| tobiash | maybe because there is no neutron there at all? | 17:42 |
| tobiash | in my cloud I think there is a neutron using provider network | 17:42 |
| tobiash | maybe that's the difference | 17:42 |
| tobiash | pabelanger: enjoy the lake :) | 17:42 |
| smcginnis | There is neutron, just no FIPs | 17:42 |
| fungi | unless it's an old enough cloud to still be using nova-network i expect neutron would be present | 17:43 |
| smcginnis | rocky, so should be good. | 17:44 |
| *** jamesmcarthur has joined #zuul | 17:45 | |
| *** jamesmcarthur has quit IRC | 17:45 | |
| *** jamesmcarthur has joined #zuul | 17:45 | |
| ofosos | Phew, finally homing in on the completion of the test cases for Bitbucket. Basic stuff is running. It's still missing the plain ref, tags, branches and enque and deque stuff. Not talking about cross driver interaction. | 17:56 |
| smcginnis | Sorry, one more question. Can I get zuul/ansible to use /usr/bin/python3 via a setting somewhere? Getting "/bin/sh: 1: /usr/bin/python: not found\\n" in the logs when it tries to set up the node for a test run. | 18:03 |
| smcginnis | I see nodepool has providers.cloud-images.python-path, but don't see anything to set ansible_python_interpreter | 18:04 |
| tobiash | smcginnis: I think the according change in zuul has not landed yet | 18:04 |
| smcginnis | tobiash: OK, so just limited to probably a setup task to make sure py2 is installed first for now? | 18:04 |
| clarkb | it was hard setting python2 becaus eansible wasn't reliable on 3 yet at the time. But latest ansible is probably much better? | 18:05 |
| tobiash | smcginnis: but as a workaround you can set it with host-vars on the job: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.host-vars | 18:05 |
| smcginnis | OK, I'll give that a shot. | 18:07 |
| pabelanger | we need some more eyes on https://review.opendev.org/637339/ for python3 support | 18:07 |
| pabelanger | tobiash: +3 | 18:07 |
| smcginnis | tobiash: Which var should I set? | 18:08 |
| tobiash | \o/ | 18:08 |
| tobiash | smcginnis: ansible_python_interpreter | 18:08 |
| smcginnis | tobiash: Cool, thanks. | 18:08 |
| pabelanger | should promote jobs be running in gate for zuul? | 18:19 |
| openstackgerrit | Merged zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 18:19 |
| pabelanger | I just noticed this failure: http://logs.openstack.org/67/664667/3/gate/opendev-promote-javascript-content/18a78ba/job-output.txt.gz | 18:19 |
| pabelanger | clarkb: corvus: tobiash: mordred: https://review.opendev.org/664006/ is break gate pipeline, can we remove +A? | 18:22 |
| pabelanger | that is opendev tarballs job | 18:22 |
| clarkb | pabelanger: it can't merge until it works aiui | 18:23 |
| pabelanger | clarkb: okay, this is the error I am seeing: http://logs.openstack.org/67/664667/3/gate/opendev-promote-javascript-content/18a78ba/job-output.txt.gz#_2019-06-12_17_51_27_575328 | 18:23 |
| clarkb | right that change hasn't merged yet | 18:24 |
| clarkb | its being used to iterate on a working pipeline | 18:24 |
| pabelanger | okay, so reading commit message, follow up is to remove | 18:25 |
| SpamapS | did we ever figure out if promote-style pipelines guarantee ordering? | 18:26 |
| corvus | pabelanger: yeah, looking at that, i think because of the way sql caching works, we might actually break the gate pipeline with that trick | 18:26 |
| SpamapS | or specifically, supercedent pipelines | 18:27 |
| pabelanger | corvus: okay, great. I didn't know it was expected result | 18:27 |
| corvus | pabelanger: so i think you're right, now that those jobs are passing, we should -1 it and re-do it without them. | 18:27 |
| fungi | SpamapS: only if the job is unique to that project+branch i believe | 18:27 |
| pabelanger | SpamapS: yes, we are using it for zuul.a.c now | 18:27 |
| corvus | pabelanger: it wasn't, i think you caught an error :) | 18:27 |
| pabelanger | Oh, yay | 18:28 |
| fungi | SpamapS: so if you have the job updating some shared external resource and triggered from multiple branches or projects, then there's no guarantee the same job won't run multiple builds concurrently and race on updating that shared resource | 18:28 |
| pabelanger | corvus: should I remove workflow? | 18:29 |
| SpamapS | fungi: even if I put a semaphore on it? | 18:29 |
| corvus | pabelanger: no | 18:29 |
| pabelanger | ack | 18:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add opendev tarball jobs https://review.opendev.org/664006 | 18:30 |
| fungi | SpamapS: i think that may solve it? i have to refresh my memory on whether semaphores are global | 18:30 |
| fungi | or at least pipeline-wise | 18:31 |
| fungi | s/wise/wide/ | 18:31 |
| SpamapS | even if they're just pipeline-local that's enough | 18:31 |
| corvus | global | 18:31 |
| pabelanger | SpamapS: fungi: yah, we use a semaphore for our promote job, with multiple projects able to trigger the job | 18:31 |
| fungi | yeah, that ought to cover the remaining race then | 18:31 |
| SpamapS | Yeah I have a semaphore and I'm about to add a 2nd repo to promote | 18:31 |
| pabelanger | ++ | 18:31 |
| SpamapS | so just wondering if that will work the way I want. | 18:31 |
| pabelanger | I believe it will | 18:31 |
| fungi | granted, using a semaphore in a regular independent pipeline likely produces the same resuly | 18:31 |
| SpamapS | supercedent should just avoid wasteful extra promotions | 18:32 |
| fungi | supercedent pipelines mostly just shine for that, right, deduplication of builds | 18:32 |
| SpamapS | also can I add a delay to supercedent pipelines? I often land 3-5 changes about 30s after each other and they currently all run promote | 18:32 |
| fungi | huh, i don't think there's a pipeline delay feature but that sounds like an interesting addition | 18:33 |
| SpamapS | yeah because it's really never hitting superceding. | 18:34 |
| SpamapS | I don't have quite the volume where I'm landing 10 - 15 commits and thus want to skip lots of promotes | 18:34 |
| SpamapS | It's just slow enough to suck | 18:35 |
| corvus | you should be able to see it in action with 3 changes | 18:35 |
| corvus | oh, unless the promotes are fast | 18:35 |
| SpamapS | https://photos.app.goo.gl/LQZRVd5SxsowWC7b8 | 18:36 |
| SpamapS | This is a very common state for us | 18:36 |
| SpamapS | Promote is 2 minutes | 18:36 |
| SpamapS | build and push is longer | 18:36 |
| corvus | SpamapS: if promote takes 2 minutes and you approve 3 changes 30s apart, you should see the middle one disappear if the pipeline is supercedent | 18:38 |
| fungi | i think if there were a pipeline delay option the way it could work is enqueue changes immediately but keep the active window at 0 changes, if creating a new queue, then increase the active window after the queue has existed for the delay period | 18:38 |
| corvus | SpamapS: (also you have a syntax-error alarm bell in the top right) | 18:38 |
| corvus | yeah, i think the option would be a good one. it would be slightly more efficient for the case SpamapS describes (at the cost of some additional time to the first promotion) | 18:40 |
| SpamapS | corvus:WHOA THAT IS COOL | 18:41 |
| SpamapS | I didn't notice it | 18:41 |
| SpamapS | I knew about the error | 18:41 |
| SpamapS | but... that's cool as hell | 18:41 |
| *** igordc has quit IRC | 18:43 | |
| fungi | you tend not to spot it because it's only there when there's an error | 18:44 |
| SpamapS | have you tried <blink> | 18:47 |
| SpamapS | ? | 18:47 |
| corvus | it's under construction :) | 18:47 |
| fungi | a barbed wire horizontal rule, spinning skulls and flaming text should make it a little more obvious | 18:48 |
| *** bhavikdbavishi has quit IRC | 18:54 | |
| SpamapS | fungi: stop stealing designs from my geocities page | 18:54 |
| tobiash | wow, I've just discovered this: https://docs.ansible.com/ansible/devel/porting_guides/porting_guide_2.8.html#jinja-undefined-values | 18:58 |
| tobiash | this alone makes it worth upgrading to ansible 2.8 :) | 18:58 |
| pabelanger | tobiash: yah, ran first 2.8 job on zuul.o.o other night, worked as expected | 19:00 |
| pabelanger | I think once I get some of these github issues under control, I'll have time to push on 2.8 default | 19:01 |
| SpamapS | yeah that's beautiful | 19:01 |
| pabelanger | SpamapS: clarkb: did we ever figure out steps for next release of zuul? cc corvus | 19:02 |
| clarkb | I'm happy to release the last commit opendev restarted on I think | 19:03 |
| tobiash | I think we should land https://review.opendev.org/637339 before the next release because the according nodepool change has already landed | 19:05 |
| tobiash | also https://review.opendev.org/661096 (squash merge) and parent would be something useful for github users (but not urgent for next release) | 19:07 |
| clarkb | my test concern was never addressed in that change fwiw | 19:10 |
| clarkb | I don't think it is the end of the world but the test is asserting the old behavior not the new one | 19:10 |
| tobiash | clarkb: the default is /usr/bin/python where the test asserts for 'python2' which is different | 19:22 |
| tobiash | Sure the result will be the same interpreter in the end but we're only asserting vor the variable value | 19:23 |
| openstackgerrit | Merged zuul/zuul master: Bump default_read_timeout for github driver to 300 https://review.opendev.org/664667 | 19:25 |
| pabelanger | clarkb: I left a comment related to your test comment, not sure if you seen that | 19:29 |
| clarkb | pabelanger: I did, I don't think your concern matters for the test suite because we control that | 19:29 |
| clarkb | (basically we'd have to require python3 for testing zuul which is already true) | 19:30 |
| clarkb | currently we require python2 and python3 | 19:30 |
| pabelanger | hmm, I guess that is right | 19:30 |
| pabelanger | however, in that context, isn't this the remote node from nodepool? I'm unsure if anything is using python3 yet for testing | 19:31 |
| pabelanger | (remote ansible I mean) | 19:31 |
| clarkb | pabelanger: its the "remote" local node in the test suite | 19:31 |
| clarkb | I think it usually sshs to localhost | 19:31 |
| pabelanger | Oh | 19:31 |
| pabelanger | for some reason I thought is was multinode | 19:31 |
| pabelanger | then, yah, I agree | 19:32 |
| pabelanger | I can push up that change and see what happens for zuul | 19:32 |
| clarkb | but tobiash is right the test sets 'python2' and the default it /usr/bin/python2 so we are checking a difference | 19:32 |
| pabelanger | k | 19:33 |
| clarkb | any reason to not approve it at this point? | 19:33 |
| pabelanger | don't think so | 19:33 |
| pabelanger | I can rebase 659812 while we are at it for more test coverage | 19:33 |
| clarkb | ok approved | 19:34 |
| pabelanger | yay | 19:34 |
| pabelanger | tobiash: are you running 661096 in production? | 19:35 |
| tobiash | pabelanger: not yet | 19:35 |
| *** jamesmcarthur has quit IRC | 19:36 | |
| pabelanger | ack | 19:36 |
| tobiash | I try to minimize the patches we run on top of master | 19:36 |
| pabelanger | +1 | 19:37 |
| tobiash | And this was not critical | 19:37 |
| pabelanger | yah, nice to have for sure. I've had potential ansible teams asking for that | 19:38 |
| pabelanger | cool to point to patch in progress | 19:38 |
| *** jamesmcarthur has joined #zuul | 19:43 | |
| Shrews | tobiash: was there a reason for not adding the +W to https://review.opendev.org/649214 yet? | 19:47 |
| corvus | weren't we going to move those jobs out of nodepool? | 19:49 |
| Shrews | corvus: i seem to remember talk of that, but no plan of action | 19:50 |
| tobiash | Shrews: I wasn't sure if you or corvus want to look at that because it's more opendev infra related | 19:51 |
| Shrews | corvus: are you suggesting a hard cut on supporting such changes... because I could get behind that. :) i hate nodepool has to test things that aren't nodepool | 19:52 |
| corvus | i don't think it needs to be a particularly sophisticated plan. i think it's wrong for those jobs to run on nodepool. i'm not going to -2 it, but i'm pretty reluctant to +2 it. | 19:53 |
| tobiash | So it's good that I didn't +w it | 19:56 |
| Shrews | corvus: what about we +3 this one, but with a comment that we'll not accept future changes. that may at least get the ball going on moving them dib | 19:56 |
| Shrews | to* dib | 19:56 |
| corvus | Shrews: wfm | 19:57 |
| *** mattw4 has quit IRC | 19:58 | |
| Shrews | ianw: i left a comment on the approval of https://review.opendev.org/649214 based on the above conversation | 20:01 |
| corvus | we're probably in a much better place to consider how to re-do those jobs in dib than the last time we talked about this | 20:01 |
| corvus | tobiash, clarkb, fungi: i'm satisfied that the promote jobs are working well enough to land this: https://review.opendev.org/664006 | 20:02 |
| corvus | (it no longer self-tests the promote jobs in gate since pabelanger discovered the problem with that, however, we did see them function in the previous patchset, so i think it's good) | 20:03 |
| Shrews | tobiash: no problem. i just wasn't sure if there was something else i wasn't considering. thx | 20:04 |
| corvus | clarkb, fungi: i'd also like to make this improvement: https://review.opendev.org/665009 but i think we can land it in any order | 20:04 |
| corvus | (you can see the result here: https://tarballs.opendev.org/zuul/zuul/ ) | 20:04 |
| fungi | ahh, so that also gets us the separate name for zuul-master-js-content.tar.gz | 20:06 |
| fungi | i wonder if extra shouldn't be before branch? | 20:07 |
| corvus | fungi: yep | 20:07 |
| fungi | though that would make it harder to implement | 20:07 |
| corvus | i think it would be easy? | 20:07 |
| corvus | and i could go either way on that | 20:08 |
| fungi | oh, maybe | 20:08 |
| fungi | thinking is that it would be more in keeping with having the artifact version in the end | 20:08 |
| corvus | just change the placement of the "name_extra" thing in "name_target" | 20:08 |
| fungi | so zuul-js-content-4.5.6.tar.gz rather than zuul-4.5.6-js-content.tar.gz | 20:08 |
| corvus | oh there's no version | 20:09 |
| corvus | see https://tarballs.opendev.org/zuul/zuul/ | 20:09 |
| fungi | right, in this case it's a branch name | 20:09 |
| corvus | ah i see what you're saying | 20:09 |
| fungi | just saying it would be *consistent* with where we'd expect to find a version in similar tarballs if published side-by-side | 20:09 |
| fungi | sorting would be weirder otherwise | 20:09 |
| corvus | though we have the "-py2-non-any" suffix at the end for the wheel | 20:10 |
| corvus | but if we had the version there, it would be earlier | 20:10 |
| corvus | fungi: our current system agrees with your inclination -- the original name of the tarball is "zuul-content-latest.tar.gz" | 20:11 |
| * corvus repaints the shed | 20:11 | |
| corvus | done | 20:12 |
| fungi | yeah, the wheel spec combines version and platform indicators, but i'd consider that all part of the version for the artifact effectively | 20:13 |
| fungi | with wheels, the expectation is you may have multiple wheels for the same version but different platforms, which is why they decided that goes after the version (so the versions sort together rather than the platforms sorting together) | 20:17 |
| corvus | makes sense | 20:19 |
| *** jamesmcarthur has quit IRC | 20:23 | |
| fungi | one other reason i can see for doing it as project-extra-branch is that if we ever decide we want to move the js content to a new zuul-js-content repo, the artifact name stays the same | 20:25 |
| fungi | perhaps a minor consistency since this job would start writing it to a different directory | 20:26 |
| pabelanger | Hmm, I just noticed zuul and nodepool don't share the same change queue. Should we do that becasue of zuul-quick-start job? | 20:30 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 20:31 |
| *** rf0lc0 has joined #zuul | 20:33 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 20:35 |
| *** rfolco has quit IRC | 20:36 | |
| *** mattw4 has joined #zuul | 20:36 | |
| corvus | tobiash, Shrews: in order to move nodepool to the zuul tenant, we would need to include devstack and all of its required projects in order for the functional jobs to work | 20:39 |
| corvus | we could either do that, or we could go ahead and rework the functional job not to inherit from devstack | 20:40 |
| openstackgerrit | Merged zuul/nodepool master: Add Debian Buster boot tests https://review.opendev.org/649214 | 20:40 |
| corvus | clarkb: ^ also | 20:40 |
| tobiash | Doesn't it need devstack in any case to be able to use a cloud? | 20:42 |
| corvus | tobiash: yes, but it doesn't need to be implemented as a plugin and inheriting from the devstack job... it could run devstack like a normal user would, or we could use OSA (which might be simpler and faster at this point) | 20:43 |
| tobiash | Ah ok | 20:44 |
| fungi | osa would be good insofar as it's also ansible | 20:44 |
| fungi | so better alignment with the tooling already in use for zuul | 20:44 |
| pabelanger | OSA would be neat, i started work on a deployment of it via zuul a few months ago. I found it pretty easy to work with | 20:46 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 20:46 |
| clarkb | I don't know that osa is faster | 20:47 |
| corvus | hrm, apparently the expected runtime of osa is 45-60m; that doesn't sound faster than devstack | 20:47 |
| clarkb | I asked about that in th ereplace devstack with osa thread and well no one wanted to tell me how much faster it was :) | 20:47 |
| corvus | doesn't it use container images? | 20:48 |
| pabelanger | lxc / systemd when I last looked | 20:48 |
| clarkb | not prebaked ones. They install into a venv in a lxc container iirc | 20:48 |
| corvus | ooooh. well, yeah, that's not advantageous then. | 20:48 |
| clarkb | so the containre is really there to isolate the deps I think | 20:48 |
| corvus | okay. kolla-ansible then? | 20:49 |
| clarkb | maybe? I don't know they are any faster either because their images are massive | 20:51 |
| clarkb | (so a lot of time is spent doing io) | 20:51 |
| corvus | clarkb: maybe io from our caching proxy is ok? | 20:51 |
| clarkb | ya I don't think its problematic other than I don't think it is actualy faster than devstack | 20:52 |
| openstackgerrit | Merged zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 20:52 |
| corvus | loci? | 20:53 |
| clarkb | loci likely is the best bet if that is our criteria | 20:53 |
| fungi | i think you still need some framework to configure and deploy as loci is just images? | 20:53 |
| fungi | not positive about that though | 20:54 |
| fungi | i know it's targeted at being much lighter-weight images at least | 20:54 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 20:54 |
| corvus | fungi: hrm. that sounds like work. | 20:56 |
| corvus | does anything out there use loci? | 20:58 |
| clarkb | hogepodge: ^ | 20:59 |
| fungi | that's who i was about to ask too ;) | 21:00 |
| corvus | clarkb: maybe git clone devstack && ./stack.sh is the way to go. | 21:02 |
| corvus | hrm... maybe we could apt-get install openstack. | 21:03 |
| clarkb | for all of devstack's problems it does address this particular use case reasonably well. I suppose we don't need to test againts future openstack though which may introduce unexpected problems (stable would be fine) | 21:03 |
| corvus | clarkb: yeah, the only thing that rubs me the wrong way about devstack is that i don't really want to install git master of everything | 21:05 |
| corvus | i'd like to just install the latest release | 21:05 |
| corvus | (after all, very few of the clouds nodepool i used with actually run git master :) | 21:06 |
| openstackgerrit | Merged zuul/zuul master: Fix test-logs.sh utility for multi-ansible https://review.opendev.org/661435 | 21:06 |
| clarkb | you can have devstack deploy stable/foo reasonably well but that is still from tip of that brnach | 21:06 |
| corvus | s/i/is/ | 21:06 |
| corvus | i'm guessing installation with packages would have the same coordination work necessary for loci | 21:07 |
| corvus | (with ubuntu packages) | 21:08 |
| clarkb | I believe zigo said the debian packages will get you a working cloud out of the box with minimal debconfing | 21:08 |
| clarkb | they use sqlite but maybe that is good enough for this use case | 21:08 |
| corvus | oh interesting | 21:09 |
| *** rf0lc0 has quit IRC | 21:09 | |
| corvus | all right, how about this: why don't i take a little bit of time and see what a plain-devstack job looks like for nodepool. most of the work there is going to be refactoring the job itself to not be a devstack plugin. once that's done, if we like it we can switch to it. and at any point, we can swap in other ways of deploying openstack (loci/debian/etc) and see if they're any faster/more reliable/etc...? | 21:11 |
| clarkb | wfm | 21:11 |
| corvus | Shrews, tobiash, fungi: ^ | 21:11 |
| clarkb | we can like have a docker compose thing that starts a nodepool after devstack is done doing its thing. Then if you swap out another deployment tool for openstack the nodepool bit remains fixed | 21:12 |
| corvus | oh interesting...i was thinking of just doing a source python install... but we could docker-compose and depend on the buildset registry | 21:14 |
| *** pcaruana|afk| has quit IRC | 21:16 | |
| corvus | why is there a "-py35" component in the functional job name? | 21:19 |
| clarkb | I think because it is running using python 3.5 on xenial | 21:20 |
| clarkb | for the services (devstack and nodepool) | 21:20 |
| corvus | there's nothing about that job which sets that though | 21:20 |
| corvus | afaict it's relying entirely on the behavior of the parent devstack job | 21:20 |
| corvus | (so if devstack switched to py36, we'd be running under py36 and our job would still be called py35) | 21:21 |
| clarkb | ya seems it only sets the USE_PYTHON3 flag to true so it will find whatever the platform python3 is | 21:22 |
| clarkb | (there is a way to set the python3 version in devstak and it will fail if it doesn't exist iirc) | 21:22 |
| corvus | clarkb: there's a -src job too, which, if we want to keep it, means we should probably install nodepool from source rather than using the images (so that we can install sdk, glean, etc from source) | 21:22 |
| clarkb | hrm I know those jobs have been really valuable on the sdk side of things (gives them test coverage of real world application) | 21:23 |
| clarkb | on the nodepool side it may be able to assume the others won't break it? | 21:23 |
| clarkb | (though that job exists because well that wasn't always the case) | 21:24 |
| fungi | non-devstack-plugin nodepool job using devstack sounds like a good stepping stone | 21:24 |
| corvus | it just occurred to me that we may want to have the image build jobs honor depends-on for things like that (i think right now if we had a nodepool change depends-on an sdk change, when we build the nodepool image, we're still just going to install sdk from pip -- we'd need something like tox-siblings for docker build) | 21:24 |
| corvus | clarkb: yeah, i'm inclined to think we should keep the job around for cross-testing nodepool-sdk changes. so i think we should concieve of it as a python-install-from-source job rather than a docker-compose thing. | 21:25 |
| corvus | (and we can still have the nodepool project as an untrusted project in the openstack tenant for the purpose of providing that job to sdk; they just won't be able to co-gate) | 21:27 |
| corvus | (and of course likewise adding sdk to the zuul tenant) | 21:27 |
| clarkb | ya they don't cogate today iirc | 21:27 |
| clarkb | so that won't be a regression but we'll still have the test coverage | 21:27 |
| corvus | yeah, we should end up in the same place as today | 21:27 |
| openstackgerrit | Merged zuul/zuul master: Remove unused code https://review.opendev.org/661436 | 21:30 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:45 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:47 |
| openstackgerrit | Merged zuul/zuul master: Increase test timeout to 6 minutes https://review.opendev.org/664961 | 21:49 |
| clarkb | thinking about cuttinga release, is the python version selector the only thing we need for that? | 21:54 |
| clarkb | if so we can likely restart the opendev zuul with that included soon (though Iwas hoping we'd catch the memory leak in action) | 21:55 |
| fungi | thread on openstack-discuss just reminded me, openstack-helm uses loci images | 21:57 |
| fungi | so maybe we could use their helm charts to configure and deploy an openstack in jobs? | 21:58 |
| clarkb | looks like it has been about 2 weeks between memory leak instances | 22:04 |
| clarkb | so our restart the other day likely has a while to go before hitting it again | 22:04 |
| fungi | so we may as well and not hold up zuul release? | 22:05 |
| fungi | and hope we catch it between this release and the next? | 22:07 |
| clarkb | ya thats what I'm thinking | 22:07 |
| corvus | when i write playbooks for jobs, i start by writing them like this: http://paste.openstack.org/show/752843/ | 22:08 |
| corvus | i like that ansible makes that an obvious way to create a process | 22:08 |
| openstackgerrit | Merged zuul/zuul master: executor: use node python path https://review.opendev.org/637339 | 22:08 |
| fungi | neat way to go about it. start with the outline | 22:09 |
| fungi | then go back and fill in the implementation details | 22:10 |
| fungi | don't even need to decide what sort of module the task will rely on | 22:10 |
| fungi | at least at first | 22:10 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:21 |
| openstackgerrit | Merged zuul/zuul master: Safely add Ansible password lookup plugin https://review.opendev.org/662870 | 22:29 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:35 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:39 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:41 |
| fungi | i suppose once we do the impending zuul restarts for opendev, i can recheck 663119 and it should hopefully work | 22:45 |
| openstackgerrit | Merged zuul/zuul master: Add opendev tarball jobs https://review.opendev.org/664006 | 22:51 |
| hogepodge | corvus: clarkb: openstack-helm uses loci | 22:58 |
| hogepodge | we've talked with keystone about running their gate jobs in a loci container | 22:59 |
| hogepodge | initial patches were sent up a while ago but we haven't followed up on it | 22:59 |
| *** jamesmcarthur has joined #zuul | 22:59 | |
| fungi | hogepodge: any feel for whether osh/loci would provide a passable (and ideally faster) means of standing up an arbitrary aio openstack for purposes of testing client/sdk interactions? | 23:01 |
| fungi | faster than devstack i mean | 23:01 |
| hogepodge | How fast is devstack? Are you including container build time? | 23:02 |
| hogepodge | I've been developing this: https://github.com/hogepodge/locistack | 23:02 |
| hogepodge | It can bring up a basic AIO in a few minutes, not accounting for container build time. | 23:03 |
| fungi | i expect we could reconsume existing loci images. we don't need openstack services from arbitrary source | 23:04 |
| clarkb | http://logs.openstack.org/16/665016/2/check/openstack-helm-compute-kit/9bd2716/job-output.txt.gz#_2019-06-12_22_04_36_607672 took about a half an hour from job start to hvaing networks and stuff configured | 23:04 |
| clarkb | which is about the same as devstack | 23:04 |
| fungi | is that also building the containers though? | 23:04 |
| fungi | er, container images | 23:05 |
| hogepodge | The build phase is distinct from the deploy phase, so a set of stable containers could be pre-built. It can take a while to do that build. 15 minutes or so for the requirements container then another 15 minutes to build the service containers in parallel | 23:06 |
| clarkb | I don't think that job is building images | 23:06 |
| fungi | if it's fetching images from somewhere, we could likely fetch the same ones | 23:07 |
| *** jamesmcarthur has quit IRC | 23:08 | |
| clarkb | the logs aren't terribly verbose about hwere it is getting things from. | 23:11 |
| *** michael-beaver has quit IRC | 23:12 | |
| hogepodge | Can you point me to the job you want to run? I can see if I can run the test against it | 23:15 |
| fungi | it's currently a devstack plugin which starts nodepool and then configures it to allocate/free server instances in the devstack-managed nova et al | 23:16 |
| fungi | work is being done to replace the devstack plugin so it doesn't have to be dependent on devstack as the deployment implementation | 23:17 |
| fungi | mostly we were investigating options for replacing devstack in that context | 23:17 |
| fungi | the goal is simply to be able to test that changes to nodepool's openstack driver continue to work with openstack | 23:18 |
| fungi | so we don't really have the same need of testing changes to openstack itself which devstack was designed for | 23:19 |
| clarkb | however at least one of the alternatives is 2-3x slower than devstack so we don't want to replace devstack with something slower | 23:19 |
| fungi | right, pretty much that | 23:20 |
| hogepodge | Let me work on some timing numbers and get them to you tomorrow afternoon. | 23:20 |
| fungi | i guess the question is whether there are more optimal solutions for quickly deploying a small, running openstack on an 8gb ram test node | 23:20 |
| fungi | so loci came up as a possible alternative | 23:21 |
| clarkb | and for a sense of the services needed, nodepool wants nova, neutron, and glance. It doesn't need cinder or swift to test what it does | 23:21 |
| clarkb | (that set implies keystone) | 23:21 |
| corvus | hogepodge: any particular shortcoming of opendev that lead you to prefer github for that? | 23:22 |
| fungi | it likely predates his involvement in loci | 23:22 |
| hogepodge | corvus: I was working on that as a poc, moving that project to opendev is in the plans | 23:23 |
| fungi | oh, i see, the locistack thing | 23:23 |
| corvus | cool, it's got a groovy ci system that would be really useful for that i think :) | 23:23 |
| hogepodge | I didn't want to create a repository that I would try ideas on then throw away (there are about three revisions of this particular project). One of the work items for loci is to do integrated testing on the containers so we can tell if they actually work. | 23:24 |
| fungi | and i guess loci itself is already in opendev | 23:24 |
| hogepodge | Yeah, this would be a new addition to it. My own time constraints related to conference travel and speaking are the main thing holding it up now. | 23:24 |
| corvus | yeah, based on what locistack says on the tin, that sounds like exactly the thing we were looking for earlier, so it'll be great to try it out :) | 23:24 |
| clarkb | fwiw each iteration could live in the same repo | 23:24 |
| corvus | hogepodge: if it shows up in opendev, i can guarantee at least one gate job will magically appear :) | 23:25 |
| fungi | yeah, locistack looks bang on... i missed the url to that somehow | 23:26 |
| fungi | clarkb: on the roster of services, i suppose we *do* need keystone too, right? | 23:27 |
| clarkb | fungi: ya that was my keystone is implied comment | 23:27 |
| fungi | see, somehow i'm failing to read but alternating comments | 23:27 |
| fungi | it must be after hours for me now | 23:27 |
| hogepodge | It's missing a lot of things, ssl termination (it was there before, but I hadn't worked out how to do proper cert management to make glance and swift happy to talk to one another, so I pulled it out) | 23:28 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 23:28 |
| clarkb | some services work without keystone but I don't think that set in conjunction with each other do | 23:28 |
| fungi | hogepodge: we've got a solution for that in opendev's jobs if you want to crib it | 23:28 |
| hogepodge | keeping secrets secret is mainly a joke, so you definitely don't want to expose it to public networks. logs will probably leak secrets | 23:28 |
| hogepodge | definitely want to crib it :-) | 23:28 |
| hogepodge | since i write everything to standard out/err the logs are pretty decent from docker-compose | 23:29 |
| corvus | we've got a role that collects all the logs from docker compose too, that's been really nice | 23:30 |
| clarkb | Shrews: tristanC left some thoughts on https://review.opendev.org/#/c/659209/3 mostly about making that code easier to understand. I don't think it is strictly necesary so let me know if you think it should go in as is | 23:37 |
| tristanC | clarkb: thanks | 23:57 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port https://review.opendev.org/659209 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!