| SpamapS | I'd be very interested in help finishing the EC2 driver. | 00:15 |
|---|---|---|
| SpamapS | The tests I wrote are 80% ... but need some focus to finish. | 00:15 |
| SpamapS | (Also would love to merge it with some kind of experimental flag since I've been running it in production for 6 months now.) | 00:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Test a child job depending on a parent that did not run https://review.openstack.org/629885 | 00:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Log connection exceptions when starting web https://review.openstack.org/630796 | 00:20 |
| pabelanger | tobiash: SpamapS have you by chance seen zuul-scheduler crash when you remove a project from a tenant? I don't think openstack has removed any yet. Just sf.io scheduler die today because OOMkiller, and think we might be leaking something when we remove a project. To late for me to debug today, but going to try and reproduce tomorrow locally. | 00:22 |
| mordred | SpamapS: I'd love for the ec2 driver to be magically finished | 00:28 |
| mordred | SpamapS: I think my only concern with landing it now is that I'm not sure how comfortable we are that we wouldn't break your production deploy | 00:29 |
| mordred | if we're not happy with the current state of testing | 00:29 |
| mordred | like - I'm sure the code as it stands today is probably fine to land ... I just don't know how to know if landing a patch that touches it will cause your entire company to be deleted | 00:30 |
| mordred | (and I don't want to delete your company) | 00:30 |
| tristanC | mordred: the driver api is not stable too, e.g. ProviderPool now needs to implement a load() method | 00:32 |
| tristanC | it seems like having the code merged would be better as unit-test would have prevent that | 00:33 |
| tristanC | otherwise, next nodepool release will break spamaps anyway | 00:33 |
| mordred | yah - and as Shrews works through cleaning up some of that - it would be really good to | 00:49 |
| mordred | gah | 00:49 |
| mordred | yeah | 00:49 |
| mordred | what you said :) | 00:49 |
| SpamapS | pabelanger: no I've never removed a project. | 01:14 |
| SpamapS | mordred: that concern is so very appreciated, but I'm already way way worse off, as I have to port it forward if I want to upgrade nodepool. | 01:15 |
| SpamapS | I might dig out some cycles to finish the tests | 01:16 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Work in Progress https://review.openstack.org/629688 | 01:37 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 corrections https://review.openstack.org/630809 | 01:37 |
| dmsimard | slice of life: I've been writing playbooks with the expectation that Zuul can run them and it's been great to be able to use "when: zuul is defined" as a condition | 01:38 |
| dmsimard | so that the playbook can leverage things prepared by zuul if they're available or prepare those things itself | 01:39 |
| SpamapS | dmsimard: I tend to try and write playbooks that do not reference the zuul variable, and then I use include_playbook to set variables from zuul into the playbook... | 01:47 |
| SpamapS | including making roles more zuul-agnostic too | 01:47 |
| SpamapS | That way if I'm in a pinch, I can use the playbook just by setting variables. | 01:47 |
| SpamapS | Though I do also often end up with a fake zuul structure for testing. | 01:48 |
| SpamapS | (Still very much want a zuul-local that does most of that for you) | 01:48 |
| SpamapS | interesting | 01:53 |
| SpamapS | I think there's a race in nodepool | 01:53 |
| SpamapS | if a server goes to "Running" too fast, it breaks the event loop that polls for them permanently. | 01:54 |
| SpamapS | (This happens in the ec2 tests I wrote because they pretend to be the API and go to running immediately) | 01:54 |
| dmsimard | SpamapS: yeah, writing them fully agnostic makes sense for real deployment things | 01:59 |
| dmsimard | this is for ara integration tests -- I want to be able to run them locally without needing zuul but using the same playbook | 02:00 |
| dmsimard | the very convenient thing that zuul does is that is prepares repos so if you're not running in zuul you need to prepare them "manually", for example | 02:00 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 corrections https://review.openstack.org/630819 | 02:02 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver WIP https://review.openstack.org/629688 | 02:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix add_host typo https://review.openstack.org/630866 | 02:22 |
| pabelanger | corvus: tristanC: tobiash: mordred: add_host appears to be broken ^. I keep getting "msg": "Adding hosts ssh with ansible_user to the inventory is prohibited" | 02:24 |
| pabelanger | -1 until I get a test up, but that is tomorrow | 02:24 |
| tristanC | pabelanger: what's the connection type? | 02:25 |
| pabelanger | undefined | 02:25 |
| pabelanger | but should be ssh | 02:25 |
| tristanC | pabelanger: and what are the args? | 02:25 |
| pabelanger | https://github.com/ansible-network/windmill-config/pull/1 | 02:26 |
| pabelanger | I don't see a testing in zuul for it, but could be missing something obvious | 02:27 |
| clarkb | we have a job that does it in project condig or system config | 02:27 |
| clarkb | it worked except that ssh wasnt set up on the re ote | 02:27 |
| pabelanger | we don't use ansible_user I think | 02:28 |
| pabelanger | wait, ansible_ssh_port is also missing a comma | 02:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix missing safe_args for add_host https://review.openstack.org/630866 | 02:33 |
| pabelanger | that fixes ansible_useransible_password and ansible_ssh_portansible_ssh_user because of missing comma | 02:34 |
| pabelanger | but ya, needs some test coverage | 02:34 |
| pabelanger | maybe a copypasta issue, IIRC that was a security issue | 02:34 |
| pabelanger | tobiash: ^ | 02:35 |
| pabelanger | EOD now | 02:35 |
| tristanC | pabelanger: that's embarrassing :-) | 02:43 |
| corvus | tristanC: i think how-to-write-a-driver docs would be great. as for the api stability -- it's an internal api; i don't think it needs to be declared stable or versioned unless and until we decide to support it externally. considering that we're happy to land any driver with tests, i don't think we should do that for a while (supporting an external api with only 3 internal users will make it more difficult | 03:31 |
| corvus | to adjust the api as we grow more drivers). | 03:31 |
| tristanC | corvus: it would make such a doc easier to write, and as driver takes time to land, having a version check to use new feature (such as the providerpool load()) would simplify rebase | 03:35 |
| tristanC | corvus: also, are receiving comments on https://review.openstack.org/535556 ? | 03:36 |
| corvus | tristanC: i'm not sure how it would make the doc easier to write. from what i've seen, most of the time it takes for a driver to land is figuring out tests. don't let past experience (where we were making drivers for new kinds of systems) mislead you into thinking that if, say, the ec2 or azure drivers were ready we would not land them. i think we would. | 03:38 |
| corvus | tristanC: tristanC thanks for the reminder on that, i'll take a look at that this week | 03:39 |
| tristanC | alright, i'll draft a how-to-write-a-driver docs with what we have now | 03:43 |
| tristanC | though it would be nice if we could aim for a stable api and avoid frequent breaking change | 03:44 |
| corvus | yes :) | 03:45 |
| tristanC | stable internal* api | 03:45 |
| *** maxamillion has joined #zuul | 03:50 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 https://review.openstack.org/630875 | 04:35 |
| tobiash | Am I correct that the ec2 driver doesn't support building images yet? | 05:21 |
| tobiash | We'll try out the ec2 driver soon too | 05:22 |
| tobiash | SpamapS: ^ | 05:22 |
| tobiash | ianw: thanks for clarification. I'll reduce to one build worker again | 05:23 |
| ianw | tobiash: reviews welcome, but it's probably not what you want to spend your time on :) | 05:25 |
| tobiash | Ok, thanks | 05:25 |
| *** chkumar|out is now known as chandankumar | 05:38 | |
| *** AJaeger has quit IRC | 07:03 | |
| *** AJaeger has joined #zuul | 07:06 | |
| SpamapS | tobiash: it does not build images, no, but I got it 80% of the way there as a POC.. it should be relatively easy... disk-image-builder produces ec2 images pretty easily, so just need an aws driver. | 07:25 |
| * SpamapS goes to bed | 07:25 | |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 07:35 |
| ianw | clarkb: centos-7-nm seemed to clean itself up ok | 07:39 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] use bindep.txt with devstack https://review.openstack.org/630890 | 07:45 |
| *** themroc has joined #zuul | 08:06 | |
| *** panda|off is now known as panda | 08:19 | |
| *** hashar has joined #zuul | 08:23 | |
| *** jpena|off is now known as jpena | 08:33 | |
| openstackgerrit | Merged openstack-infra/zuul master: Log connection exceptions when starting web https://review.openstack.org/630796 | 08:42 |
| *** avass has joined #zuul | 08:55 | |
| *** gtema has joined #zuul | 09:20 | |
| *** pcaruana has joined #zuul | 09:30 | |
| *** bjackman has joined #zuul | 09:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use urljoin for git driver getGitUrl https://review.openstack.org/630920 | 10:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use urljoin for git driver getGitUrl https://review.openstack.org/630920 | 10:05 |
| mordred | corvus, tobiash: ^^ found a fun one ... I'm a little concerned that the job hangs indefinitely in the case of this config issue, but I think it's easy enough to prevent the issue in this context - and we really do expect zuul to be able to clone source repos | 10:06 |
| mordred | but maybe we need to ultimately trigger a job failure with a message about inability to clone - like, maybe if someone renamed a git repo out from under zuul and forgot to update their zuul config a similar scenario could happen? | 10:08 |
| mordred | I didn't dig all the way into that path yet | 10:08 |
| *** chandankumar has quit IRC | 10:31 | |
| *** arxcruz|ruck has quit IRC | 10:34 | |
| *** chandan_kumar has joined #zuul | 10:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: properly format error exception https://review.openstack.org/630928 | 10:38 |
| *** dkehn has quit IRC | 10:45 | |
| *** electrofelix has joined #zuul | 10:48 | |
| *** arxcruz has joined #zuul | 11:02 | |
| *** arxcruz has quit IRC | 11:02 | |
| *** arxcruz has joined #zuul | 11:02 | |
| *** openstackgerrit has quit IRC | 11:22 | |
| *** bjackman has quit IRC | 11:23 | |
| *** bjackman has joined #zuul | 11:28 | |
| *** openstackgerrit has joined #zuul | 11:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: runner: add execute sub-command https://review.openstack.org/630944 | 11:39 |
| tristanC | got zuul-runner execute a job locally, now failing on upload-logs missing the ssh key \o/ | 11:40 |
| tristanC | this could be fixed by a secrets substitution map provided by the user, but perhaps zuul-runner could provide a copy of the upload-logs role that doesn't need an ssh key, what do you think? | 11:41 |
| tobiash | mordred: maybe it's not as easy as it looked because the url also can be just a local path. so you might want to switch between os.path.join and urljoin depending on the start of the string | 11:43 |
| mordred | tobiash: ah - indeed | 11:46 |
| mordred | tobiash: hrm. actually - os.path.join does the right thing in all of the scenarios | 11:51 |
| mordred | tobiash: so maybe the actual fix isn't to switch to urllib.parse.urljoin but instead to switch to os.path.join like the other method | 11:51 |
| odyssey4me | Hi folks - we're seeing something which strikes me as a potential bug, so I wonder if we could walk it through to validate it or not. Essentially we're seeing nodepool nodes marked as deleting which are not present in the cloud, but in the log we see nodepool still trying to delete them, failing and re-adding them for deletion over and over again. | 11:52 |
| odyssey4me | It is possible that this has been fixed - we're using around a 6 month old version of code. | 11:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use os.path.join for git driver getGitUrl https://review.openstack.org/630920 | 11:52 |
| mordred | odyssey4me: I'm not 100% sure - but I know Shrews has done a bunch of cleanup things in that general area in the last 6 months - he might have a better idea though | 11:53 |
| tristanC | odyssey4me: perhaps https://review.openstack.org/576288 ? | 11:56 |
| odyssey4me | tristanC I thought that, but I don't think so - the log exposes this: http://paste.openstack.org/show/PntmQwB5BwEYtSWsGCs5/ | 11:58 |
| odyssey4me | so basically it's in the list for deleting, it tries to delete, times out, and reschedules the deletion to the provider again | 11:59 |
| odyssey4me | somehow it's not noticing that the provider no longer has that uuid present | 11:59 |
| odyssey4me | this is rax pub cloud, in case that matters - I know there is some funky business done there | 12:00 |
| *** jpena is now known as jpena|lunch | 12:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 12:06 |
| mordred | odyssey4me: well, luckily enough for you, we have many nodes on rax public - so definitely have the same fun :) | 12:12 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: runner: add execute sub-command https://review.openstack.org/630944 | 12:14 |
| *** gtema has quit IRC | 12:16 | |
| odyssey4me | mordred if the openstacksdk is asked to delete a node that doesn't exist - what does it return? | 12:18 |
| mordred | odyssey4me: None | 12:19 |
| mordred | odyssey4me: that is considered success by the sdk | 12:20 |
| mordred | odyssey4me: because the node doesn't exist - so thus your request has already been satisfied! | 12:20 |
| odyssey4me | in our case, openstacksdk==0.12.0 | 12:20 |
| mordred | odyssey4me: I assume rax public is working for you otherwise? | 12:21 |
| mordred | zomg. that's SO OLD | 12:21 |
| mordred | but yes - that has been shade/sdk behavior since the beginning, so you should still be fine | 12:21 |
| odyssey4me | heh, yeah - that's the min version satisfying nodepool in our case | 12:21 |
| mordred | :) | 12:21 |
| mordred | well, it has been shade behavior since the beginning - but in any case, you should be ok | 12:21 |
| *** panda is now known as panda|lunch | 12:47 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul-jobs master: WIP: Default private_ipv4 to use public_ipv4 address when null https://review.openstack.org/623294 | 12:50 |
| *** chandan_kumar has quit IRC | 12:58 | |
| *** jpena|lunch is now known as jpena | 12:59 | |
| *** chandan_kumar has joined #zuul | 12:59 | |
| *** bjackman_ has joined #zuul | 13:04 | |
| *** bjackman has quit IRC | 13:05 | |
| tobiash | mordred: if you have time for an easy review: https://review.openstack.org/630118 | 13:11 |
| *** rlandy has joined #zuul | 13:19 | |
| *** arxcruz is now known as arxcruz|ruck | 13:21 | |
| *** bjackman_ has quit IRC | 13:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 13:29 |
| *** gtema has joined #zuul | 13:32 | |
| Shrews | odyssey4me: mordred: current nodepool code has a get_server() call that checks for None and should prevent the timeout. I *think* that's always been there? So either that instance actually exists, or the sdk is lying. Otherwise, not sure what's happening | 13:34 |
| Shrews | for reference: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/provider.py#n373 | 13:35 |
| Shrews | which just calls sdk's get_server() | 13:36 |
| tristanC | "zuul-runner -a http://localhost:9000/api/ --tenant local --pipeline check --project sf-jobs --job linters execute" outputs: http://paste.openstack.org/show/742638/ | 13:38 |
| *** panda|lunch is now known as panda | 13:43 | |
| odyssey4me | Shrews yep, same code in the SHA we're using for nodepool | 13:51 |
| mordred | tobiash: wow. what a fun problem to have | 13:51 |
| odyssey4me | I'm suspecting that either we have a bad mix of libraries at play here, because we're seeing the same behaviour for a standard openstack cloud. | 13:51 |
| odyssey4me | well, actually in the openstack cloud the instances are actually there, but the delete doesn't seem to be getting to them from the launcher | 13:55 |
| odyssey4me | so I'm still suspecting a broken host of some sort | 13:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix missing safe_args for add_host https://review.openstack.org/630866 | 13:56 |
| pabelanger | tobiash: tristanC: mordred: corvus: ^add_host fix updated to include tests now | 13:56 |
| sshnaidm | pabelanger, tristanC, clarkb is that all nodepool tests we have? https://github.com/openstack-infra/zuul/blob/0b00c4685b1883dcb6d9ac1814acc68c89eeae26/tests/nodepool/test_nodepool_integration.py | 13:59 |
| sshnaidm | and where does this test run? https://github.com/openstack-infra/nodepool/blob/master/devstack/plugin.sh | 14:00 |
| sshnaidm | pabelanger, I'd like to add tests to https://review.openstack.org/630649 but not sure where | 14:01 |
| pabelanger | sshnaidm: that is the devstack funcation test for nodepool | 14:01 |
| tobiash | mordred, pabelanger, clarkb: didn't someone mention something with blocked github requests and timeout here in the last weeks? | 14:01 |
| sshnaidm | pabelanger, where does it run? | 14:01 |
| tobiash | I've a blocked zuul waiting for data of github in prod :/ | 14:01 |
| pabelanger | sshnaidm: http://git.zuul-ci.org/cgit/nodepool/tree/.zuul.yaml#n62 | 14:02 |
| pabelanger | tobiash: I haven't see that myself | 14:02 |
| tobiash | this is the stack of the blocked thread: http://paste.openstack.org/show/742642/ | 14:02 |
| mordred | tobiash: I seem to remember that - but I don't remembr what it was | 14:02 |
| sshnaidm | pabelanger, ok, I added there a test, is it enough? | 14:02 |
| mordred | tobiash: I think something about exposing the ability to configure a timeout in the github library to pass down into requests | 14:03 |
| mordred | tobiash: and I think jlk had a pr up to do that perhaps? | 14:03 |
| tobiash | mordred: we probably could attach a timeout to the session object | 14:04 |
| pabelanger | sshnaidm: you could do that, or update existing unit testing with fake-provider and validate we are passing into: http://git.zuul-ci.org/cgit/nodepool/tree/nodepool/tests/unit/test_launcher.py#n477 | 14:04 |
| mordred | tobiash: https://github.com/sigmavirus24/github3.py/pull/904 | 14:04 |
| mordred | tobiash: it has been merged | 14:04 |
| tobiash | mordred: thanks, checking if it has been released already | 14:05 |
| mordred | tobiash: I'm excited to see that the checks api pr seems to be getting close | 14:06 |
| tobiash | yes :) | 14:08 |
| tobiash | thanks to swest | 14:08 |
| tobiash | mordred: that fix is unreleased :( | 14:09 |
| mordred | ++ swest | 14:09 |
| mordred | tobiash: :( | 14:09 |
| *** gtema has quit IRC | 14:18 | |
| *** gtema has joined #zuul | 14:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 14:40 |
| tobiash | jlk: any chance to get a github3.py release soon? | 14:43 |
| *** quiquell is now known as quiquell|off | 15:07 | |
| tobiash | mordred: I finally had to restart the scheduler | 15:16 |
| mordred | tobiash: :( | 15:16 |
| tobiash | but it was fun to get from docker to network namespace to active connections to the file descriptor of that connection to getting a gdb as privileged hostpid container running on the same openshift node to attach a gdp to the scheduler to close that fd | 15:17 |
| tobiash | and I finally managed to close it but that made zuul unhappy | 15:17 |
| tobiash | -> restart | 15:18 |
| sshnaidm | pabelanger, any ideas maybe where should I fix it? http://logs.openstack.org/49/630649/4/check/tox-py35/7286d1c/job-output.txt.gz#_2019-01-15_15_01_04_640088 | 15:44 |
| pabelanger | sshnaidm: you need to add the userdata field into voluptuous schema: http://git.zuul-ci.org/cgit/nodepool/tree/nodepool/driver/openstack/config.py#n347 | 15:47 |
| mordred | tobiash: aww. I'm sad all that effort didn't solve it for you | 15:49 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 15:57 |
| clarkb | sshnaidm: not sure if you saw my note on ps4 about docs, but otherwise that is looking like it should work | 16:00 |
| *** rcarrillocruz has joined #zuul | 16:08 | |
| *** gtema has quit IRC | 16:16 | |
| *** rcarrillocruz has quit IRC | 16:18 | |
| *** hashar has quit IRC | 16:19 | |
| *** themroc has quit IRC | 16:29 | |
| *** avass has quit IRC | 16:32 | |
| *** rcarrillocruz has joined #zuul | 16:40 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use os.path.join for git driver getGitUrl https://review.openstack.org/630920 | 16:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Allow known_hosts to be run in untrusted context https://review.openstack.org/631020 | 16:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 16:48 |
| pabelanger | mordred: mind looking over ^631020 that is a follow up to get add_host working in untrusted jobs. | 16:49 |
| pabelanger | tobiash: maybe you can review also^ | 16:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 16:56 |
| *** hashar has joined #zuul | 17:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 17:18 |
| Shrews | Is John Studarus in channel??? | 17:20 |
| clarkb | Shrews: nick is studarus iirc, doesn't look like he is here | 17:21 |
| pabelanger | might have to ask him to join, I know he doesn't have an IRC proxy setup | 17:21 |
| Shrews | i think we need to reorient his gerrit workflow | 17:22 |
| Shrews | seems to be adding new reviews to correct problems with his initial baremetal driver review | 17:23 |
| pabelanger | mordred: do you have any insights into this failure with tox -notests: http://logs.openstack.org/20/631020/1/check/tox-py35/278489f/job-output.txt.gz#_2019-01-15_17_01_22_839674 | 17:24 |
| pabelanger | I guess we didn't download yarn bits properly? | 17:24 |
| mordred | pabelanger: http://logs.openstack.org/20/631020/1/check/tox-py35/278489f/tox/py35-1.log | 17:26 |
| mordred | pabelanger: or maybe something isn't providing react-scripts anymore? | 17:26 |
| pabelanger | k, will look more | 17:28 |
| pabelanger | mordred: corvus: clarkb: tobiash: tristanC: https://review.openstack.org/#/q/topic:untrusted-add_host is ready for review, if you don't mind adding to your pipeline. This should finally allow untrusted jobs to use add_host / known_hosts | 17:29 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: pep8 https://review.openstack.org/630875 | 17:35 |
| Shrews | fyi, i just left a comment for John on his original review about the pep8 reviews, e.g. ^^^ | 17:36 |
| corvus | Shrews: ++ | 17:38 |
| *** hashar has quit IRC | 17:57 | |
| *** hashar has joined #zuul | 17:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: DNM Testing fetch-output and merge-output-to-logs https://review.openstack.org/628668 | 17:59 |
| *** jpena is now known as jpena|off | 17:59 | |
| *** hashar has quit IRC | 18:00 | |
| *** hashar has joined #zuul | 18:01 | |
| *** pcaruana has quit IRC | 18:01 | |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 18:13 |
| *** studarus has joined #zuul | 18:15 | |
| * Shrews notes a wild studarus among us | 18:16 | |
| Shrews | welcome studarus | 18:17 |
| Shrews | we don't bite (well, mordred might, but he's had his shots) | 18:17 |
| studarus | yes - I'm trying to catchup on my OpenStack work. I have some driver code to checkin. I also have a new OSA cloud to finish getting up and running for the infra team. | 18:17 |
| Shrews | \o/ | 18:18 |
| *** panda is now known as panda|off | 18:19 | |
| mordred | yay | 18:20 |
| * fungi is excited to drive a new studaruscloud around the block | 18:25 | |
| *** hashar has quit IRC | 18:30 | |
| *** hashar has joined #zuul | 18:30 | |
| studarus | ha ha | 18:33 |
| studarus | my night job as a cloud operator | 18:33 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul-jobs master: install-nodejs: add support for RPM-based OSes https://review.openstack.org/631049 | 18:52 |
| *** hashar has quit IRC | 18:52 | |
| studarus | should I be watching zuul.openstack.org or is there a different one for zuul commits? | 19:10 |
| Shrews | studarus: that's the right place | 19:11 |
| electrofelix | With zuul v2 with the bonnyci patches applied for Github support, we've noticed performance issues over time where the zuul-merger processing said repos gets excessively slower over time | 19:12 |
| electrofelix | it appears most of the time is spent buried in the _iter_packed_refs and may be linked to lots of branches getting created within the cached repo for the zuul merger | 19:13 |
| electrofelix | resulting in subsequent iterations taking longer each time | 19:13 |
| electrofelix | wondering if anything like that has been seen in zuul v3? | 19:13 |
| *** rlandy is now known as rlandy|brb | 19:14 | |
| electrofelix | I've done a simple prune of the remote refs that have been deleted (couple of thousand) and it appears to have brought performance back from taking nearly 3 minutes for the repo.create_head() call in the Repo.reset() method in zuul/merger/merger.py down to ~ 3-5 seconds | 19:14 |
| SpamapS | electrofelix: that code has seen a ton of change since the bonnyci days. | 19:16 |
| SpamapS | electrofelix: one suggestion.. make people work on forks. Stop putting branches on the repos that zuul watches. :) | 19:16 |
| electrofelix | SpamapS: I wish I could do that... :-( | 19:17 |
| SpamapS | Why can't you? thousands of branches? Who is served by that? | 19:17 |
| * SpamapS has won this fight every time. | 19:17 | |
| SpamapS | which is.. twice. ;) | 19:17 |
| pabelanger | does exclude-unprotected-branches: True work here? | 19:18 |
| SpamapS | nope | 19:19 |
| SpamapS | because that just makes zuul choose not to act | 19:19 |
| SpamapS | it still has to merge the refs to decide that. | 19:19 |
| SpamapS | (IIRC) | 19:19 |
| pabelanger | ah, right | 19:19 |
| SpamapS | It's possible that it would help a bit. | 19:19 |
| SpamapS | But also he's on v2 | 19:19 |
| SpamapS | which does not have such a thing | 19:20 |
| SpamapS | electrofelix: any chance you will get off v2? | 19:20 |
| SpamapS | since it's like, you know, unsupported and everybody told us never to use it (sorry everybody, you were right ;) | 19:20 |
| electrofelix | SpamapS: I'd love to, but persuading people that moving stuff that doesn't have immediate visible improvements to users is difficult to just getting us to nodepool is hard never mind getting on to zuulv3 | 19:21 |
| electrofelix | SpamapS: unfortunately I'm not allowed mandate how other developers user their Github repos other wise this wouldn't have occurred in the first place | 19:22 |
| electrofelix | I'm not in favour of allowing branch push directly into the main github repo either | 19:22 |
| pabelanger | github doesn't have an ACL to stop it either right? unless you remove the user as committer | 19:23 |
| SpamapS | github does, but you have to deny write entirely. | 19:25 |
| SpamapS | oh yeah you said that ;) | 19:25 |
| mordred | electrofelix: I love how people don't like letting experts, you know, be the experts | 19:25 |
| SpamapS | and actually, that's the right thing to do. Kick everybody out of write access, make zuul own everything, people can work in forks. But there's one stupid thing about that: GitHub won't let you put labels on stuff unless you have write to the repo. | 19:26 |
| SpamapS | so you'll have to like, set up a utility pipeline that takes commands as comments or something. It's really.. really stupid. | 19:27 |
| electrofelix | mordred: true, but in this case I think we probaby need to come up with something that cleans up stray references anyway just as a good thing to do | 19:31 |
| electrofelix | was hoping that maybe zuulv3 had already hit this and someone might know if there were calls to do the cleaning | 19:31 |
| *** rlandy|brb is now known as rlandy | 19:33 | |
| sshnaidm | pabelanger, seems like it didn't work, node hasn't attribute "userdata": https://review.openstack.org/#/c/630649/6/nodepool/tests/unit/test_launcher.py | 19:40 |
| sshnaidm | pabelanger, do you know how can I "extract" it if I configure it so? https://review.openstack.org/#/c/630649/6/nodepool/tests/fixtures/node.yaml | 19:41 |
| studarus | is there a preferred way to add imported libraries for nodepool drivers? Zuul needs them to build the driver. | 19:41 |
| clarkb | studarus: requirements.txt for python libs and bindep.txt for system level deps | 19:42 |
| studarus | can I put in a requirements.txt in the driver directory? Or update the project one? | 19:43 |
| clarkb | update the project one | 19:44 |
| studarus | thanks | 19:44 |
| clarkb | I thnik the intent is that these plugisn actually live in tree | 19:44 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 19:46 |
| *** electrofelix has quit IRC | 19:46 | |
| pabelanger | sshnaidm: ah, right, that is the node we get from zookeeper, which won't have that data. Maybe switch to new test like test_node_boot_from_volume but with your userdata value, and confirm we can boot the node. | 19:50 |
| pabelanger | sshnaidm: the devstack test is likely the best, since we are booting image in the cloud | 19:50 |
| sshnaidm | pabelanger, it doesn't work because userdata requires cloud-init to be installed, and I suppose it's not | 19:50 |
| sshnaidm | pabelanger, but it proves it doesn't break anything | 19:51 |
| pabelanger | sshnaidm: does it show up in openstack server show foo | 19:51 |
| pabelanger | as long as there, I think that is enough for a test | 19:51 |
| pabelanger | we don't actually care that cloud-init worked | 19:51 |
| sshnaidm | pabelanger, no | 19:51 |
| clarkb | pabelanger: ya I wouldn't test it by confirming cloud-init did things. I would just check the api data on the instance | 19:52 |
| pabelanger | yah, I just don't know how to do that with userdata. need to look at docs | 19:53 |
| sshnaidm | pabelanger, it shows in create_args for nova | 19:54 |
| sshnaidm | pabelanger, but "openstack show server" won't show it, I think because it may contain sensitive data | 19:55 |
| sshnaidm | is there a way to check create_args for nova..? | 19:55 |
| clarkb | sshnaidm: pabelanger it will if you are admin (whcih we have access to in that test env) | 19:55 |
| pabelanger | yah, that's what I figured | 19:56 |
| sshnaidm | ok, will check it then | 19:57 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 20:06 |
| sshnaidm | clarkb, are nodepool nodes created in plugin.sh? I'm looking for place to execute "openstack server show" | 20:09 |
| clarkb | sshnaidm: no the running nodepool daemon creates them then our test polls for them to come up | 20:11 |
| clarkb | in tools/test_something | 20:11 |
| *** hashar has joined #zuul | 20:12 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 20:19 |
| *** studarus has quit IRC | 20:21 | |
| *** hashar has quit IRC | 20:25 | |
| *** rfolco has quit IRC | 20:28 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 20:32 |
| pabelanger | corvus: Shrews: tobiash: clarkb: are we still on track to tag next nodepool release with f8bf? Not sure where we left the discussion from yesterday | 20:35 |
| corvus | pabelanger: oh! sorry i got distracted; i'll do that now | 20:35 |
| pabelanger | thanks! | 20:35 |
| openstackgerrit | John Studarus proposed openstack-infra/nodepool master: Packet Bare Metal Nodepool Driver https://review.openstack.org/629688 | 20:36 |
| corvus | pushed 3.4.0 | 20:36 |
| pabelanger | tyty | 20:37 |
| fungi | clarkb: when i try to run `pip freeze` on the new storyboard-dev, it complains about "locale.Error: unsupported locale setting" | 20:39 |
| fungi | i wonder if we're doing something strange with our locale configuration recently | 20:39 |
| fungi | what's especially strange is that the output of the `locale` command matches the old server | 20:40 |
| fungi | oops, wrong channel! sorry zuulites :/ | 20:41 |
| *** studarus has joined #zuul | 20:58 | |
| studarus | somehow I managed to break test_delete_now... | 20:58 |
| studarus | Shrews ^ looks like I'm hitting the race condition that you have commented | 20:59 |
| Shrews | studarus: yeah, that one hits us rarely | 21:02 |
| studarus | so I have Zuul rerun? | 21:03 |
| Shrews | studarus: you can leave a 'recheck' comment, but you might want to wait to see the results of all your tests first | 21:04 |
| studarus | quickstart failed too - example user wasn't found in Gerrit... | 21:09 |
| studarus | or rather - tenant wasn't available | 21:13 |
| Shrews | studarus: that one has been a bit flakey as well | 21:15 |
| studarus | just waiting on some non-voting ones... | 21:16 |
| Shrews | studarus: you will need to add tests of your own :) | 21:16 |
| Shrews | studarus: unit tests, at least. i'm not sure how to do functional tests for that driver | 21:17 |
| studarus | yeah - I need to think up some simple ones | 21:17 |
| studarus | maybe a config file that has zero capacity | 21:22 |
| Shrews | studarus: left you a few comments after a quick glance at the current code | 21:27 |
| studarus | yes - I saw. Thanks! | 21:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add docker image promotion roles https://review.openstack.org/631078 | 22:03 |
| *** studarus has quit IRC | 22:16 | |
| *** rlandy has quit IRC | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!