| pabelanger | nhicher: tristanC: can somebody check zuul scheduler debug logs, showing the last reconfigure of zuul was: | 00:15 |
|---|---|---|
| pabelanger | Last reconfigured: Wed Sep 19 2018 05:18:35 GMT-0400 (Eastern Daylight Time) | 00:15 |
| pabelanger | but, for ansible-network we've landed changes to our project stanza | 00:16 |
| pabelanger | so, would have expected zuul to be reconfigured today | 00:16 |
| pabelanger | https://github.com/ansible-network/zuul-config/pull/69 | 00:17 |
| *** dmsimard has joined #softwarefactory | 00:20 | |
| nhicher | pabelanger: I only found reference to ansible-network/cloud_vpn#45 in scheduler.log | 00:26 |
| nhicher | https://github.com/ansible-network/zuul-config/pull/45 | 00:27 |
| nhicher | we will check with tristanC | 00:27 |
| pabelanger | nhicher: let me check the syntax to grep for | 00:29 |
| pabelanger | I think it is 'Full reconfiguration complete' | 00:31 |
| pabelanger | you'll see Full reconfiguration beginning | 00:31 |
| pabelanger | that is the start | 00:31 |
| pabelanger | but cannot remember if that is trigger'd by project stanza changes | 00:32 |
| pabelanger | for just tenant configuration | 00:32 |
| pabelanger | Yah, maybe I am confusing it, I think it might be 'Tenant reconfiguration complete' | 00:32 |
| pabelanger | but if so, I'm unsure why that PR didn't fix an issue I and working with | 00:33 |
| nhicher | pabelanger: last tenant reconfiguration was: 2018-09-21 00:24:00,828 utc, I will check for error | 00:35 |
| pabelanger | okay, yah. there must be something in logs | 00:35 |
| pabelanger | or I think there will be | 00:35 |
| pabelanger | also, I wonder if we should expose tenant reconfiguration in https://ansible-network.softwarefactory-project.io/zuul/api/status too | 00:36 |
| pabelanger | so we can look to status page for each tenant | 00:36 |
| nhicher | pabelanger: yes, it's probably easier for user: 151M /var/log/zuul/scheduler.log | 00:38 |
| pabelanger | yah, gets verbose | 00:38 |
| pabelanger | if gerrit still set to debug level? | 00:38 |
| pabelanger | that should be set to INFO, to help reduce logs | 00:38 |
| nhicher | yes, gerrit level is debug | 00:40 |
| pabelanger | yah, that will take up a lot | 00:42 |
| nhicher | but we have to restart the scheduler to apply | 00:42 |
| pabelanger | yah, I'd love to get logging to dynamically updated some how | 00:43 |
| pabelanger | either via HUP or some other signal | 00:43 |
| nhicher | pabelanger: I don't find anything on log files, we have to wait for tristanC or fbo | 00:59 |
| nhicher | pabelanger: found error 'ERROR Project github.com/ansible-network/network-engine does not have the default branch master | 01:00 |
| nhicher | but should not be related to your change | 01:00 |
| pabelanger | ack, I'll fix network-engine, thanks | 01:01 |
| pabelanger | okay, lets see what tristanC says | 01:01 |
| pabelanger | another option is to run a full-reconfigure manually via CLI and see if that works. But that doesn't tell us why I'm still seeing default branch master error on cloud_vpn project | 01:02 |
| nhicher | pabelanger: do you know any project on zuul upstream which use default-branch stanza ? | 01:04 |
| nhicher | I did not find anything on openstack.org/zuul | 01:04 |
| pabelanger | nhicher: no, but rdoproject does | 01:05 |
| pabelanger | everything upstream is master | 01:05 |
| nhicher | pabelanger: ok, good to know | 01:05 |
| pabelanger | but ansible is devel and rdo distgit is rpm-master | 01:05 |
| pabelanger | long term, with zuul, they likely should switch back to master | 01:05 |
| pabelanger | but, that is work | 01:05 |
| tristanC | o/ | 01:06 |
| pabelanger | nhicher: for now, I can test with another PR, so we can deal with default-branch later. The update to openstacksdk would be better :) | 01:06 |
| nhicher | pabelanger: yes, we will update it with tristanC =) | 01:06 |
| pabelanger | great | 01:07 |
| tristanC | pabelanger: how openstack-infra manage such requirements, openstacksdk is not listed in zuul requirements.txt | 01:08 |
| pabelanger | tristanC: it is extra for pip | 01:09 |
| pabelanger | https://git.zuul-ci.org/cgit/zuul/tree/setup.cfg#n55 | 01:09 |
| pabelanger | so | 01:09 |
| pabelanger | pip install zuul[zuul_exector] | 01:09 |
| pabelanger | I think | 01:09 |
| pabelanger | ha, but we don't do that in puppet-zuul | 01:10 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/executor.pp | 01:11 |
| pabelanger | that likely should be fixed | 01:11 |
| pabelanger | but, with container work upstream, puppet-zuul is doing away soon | 01:11 |
| pabelanger | okay, wow, here is something I didn't know | 01:16 |
| pabelanger | for github PR | 01:16 |
| pabelanger | create new PR for project | 01:16 |
| pabelanger | then ammend commit | 01:17 |
| pabelanger | push up | 01:17 |
| pabelanger | but with the ammmed, add depends-on header | 01:17 |
| pabelanger | zuul will not pick up depends-on header, for 2nd PS for PR | 01:17 |
| pabelanger | you need to into web, and manual edit the first comment, and add that | 01:17 |
| pabelanger | for depends-on to work | 01:17 |
| pabelanger | I don't know why | 01:18 |
| tristanC | pabelanger: we are updating executors with 0.17.2 | 01:29 |
| tristanC | openstacksdk-0.17.2 | 01:29 |
| tristanC | pabelanger: when you required-projects network-engine, could you try using "override-checkout: devel | 01:34 |
| tristanC | " ? | 01:34 |
| tristanC | e.g. https://github.com/ansible/zuul-jobs/blob/c783dc05e3f9e17602c78fe90ed871588947a55a/zuul.d/jobs.yaml#L37-L39 | 01:34 |
| pabelanger | tristanC: yup, agree | 01:49 |
| pabelanger | I'd also expect: https://github.com/ansible-network/zuul-config/pull/69 to work | 01:49 |
| pabelanger | if you want to ping when executors are updated, I'll give swift upload a go again | 01:50 |
| tristanC | pabelanger: they are updated now | 01:53 |
| pabelanger | cool, let me try agian | 01:54 |
| pabelanger | tristanC: hmm, didn't work | 01:57 |
| pabelanger | finger://ze02.softwarefactory-project.io:7979/b7a6582ab3404f0c8e4ccf01411b32cb | 01:57 |
| pabelanger | do you have a moment to check? | 01:57 |
| tristanC | nothing in logs... | 02:14 |
| tristanC | perhaps make the swift upload ignore_error and keep the regular scp ? | 02:16 |
| pabelanger | yah | 02:20 |
| pabelanger | I'll do that in morning | 02:21 |
| *** sshnaidm|pto has joined #softwarefactory | 04:34 | |
| gundalow | tristanC: is the an example/mockup of https://review.openstack.org/#/c/599472/ available to look at? | 05:53 |
| *** sshnaidm|pto has quit IRC | 06:06 | |
| tristanC | gundalow: i've a poc deployment connected to sf-project zuul here: http://38.145.32.112/react-zuul/ | 06:15 |
| tristanC | gundalow: and you can see the /status/change route in effect here for example: http://38.145.32.112/react-zuul/t/rdoproject.org/status/change/601634,2 | 06:16 |
| tristanC | or http://38.145.32.112/react-zuul/t/rdoproject.org/status/change/603406,4 (this one has more job running) | 06:16 |
| tristanC | gundalow: also, here is the new job page: http://38.145.32.112/react-zuul/t/ansible-network/job/cloud-vpn-aws-csr-to-aws-vpn | 06:17 |
| *** chkumar|off is now known as chkumar|ruck | 06:23 | |
| *** zoli is now known as zoli|wfh | 06:25 | |
| *** zoli|wfh is now known as zoli | 06:26 | |
| *** jpena|off is now known as jpena | 07:48 | |
| sfbender | Tristan de Cacqueray created scl/zuul-jobs-distgit master: Bump version https://softwarefactory-project.io/r/13712 | 08:57 |
| sfbender | Merged DLRN master: Fix commit id in database when using rhpkg https://softwarefactory-project.io/r/13444 | 09:30 |
| sfbender | Merged DLRN master: Additional requirements for the downstream repo https://softwarefactory-project.io/r/13431 | 09:31 |
| sfbender | Merged software-factory/sf-config master: ara: fix for 1.16.1 https://softwarefactory-project.io/r/13707 | 09:57 |
| *** zoli is now known as zoli|lunh | 10:22 | |
| *** zoli|lunh is now known as zoli|lunch | 10:22 | |
| *** zoli|lunch is now known as zoli | 11:00 | |
| *** zoli is now known as zoli|wfh | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:23 | |
| *** chkumar|ruck is now known as chkumar|off | 11:46 | |
| gundalow | pabelanger: Getting `RETRY_LIMIT` and the job failing in `prepare-workspace` https://github.com/ansible-network/yang/pull/16#issuecomment-423498092 | 11:54 |
| *** jpena|lunch is now known as jpena | 12:26 | |
| sfbender | Merged scl/zuul-jobs-distgit master: Bump version https://softwarefactory-project.io/r/13712 | 12:30 |
| *** shanemcd has joined #softwarefactory | 12:53 | |
| shanemcd | o/ | 12:53 |
| matburt | \o | 13:00 |
| tristanC | leaving for the week end, have a good one folks, talk to you on monday! | 13:03 |
| pabelanger | gundalow: I'll look shortly | 13:04 |
| tristanC | shanemcd: matburt: the kibana worker was stuck, i just restarted it and it is now indexing the backlog, if you look at the dashboard you'll see job's indexes: https://ansible.softwarefactory-project.io/app/kibana | 13:09 |
| shanemcd | nice! | 13:09 |
| matburt | so hot | 13:17 |
| pabelanger | gundalow: if I had to guess, it is becasue ansible/ansible has a symlink setup inside the project. Zuul won't clone a project with symlinks because it is a potential security issue | 13:26 |
| pabelanger | but just a guess right now | 13:26 |
| gundalow | ooooooooh, nice | 13:27 |
| pabelanger | We'll have to have somebody look at executor logs, will do that this morning | 13:30 |
| pabelanger | mhu: fbo: nhicher: are you able to check ze02 and see if there is any more traceback for https://ansible-network.softwarefactory-project.io/logs/an/24/24/4a0420d708997e0c9067c0428f8cb33a4930ccfe/check/ansible-test-sanity/1ad120c/job-output.txt.gz#_2018-09-21_13_22_48_178655 | 13:36 |
| pabelanger | getting outside workdir error message from zuul | 13:37 |
| pabelanger | nhicher: are you also able to test swift upload again from executor? Still dealing with auth issue | 13:58 |
| nhicher | pabelanger: I finish a meeting and I will check | 14:05 |
| pabelanger | thanks | 14:06 |
| nhicher | pabelanger: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication | 14:21 |
| nhicher | rh-python35-python-openstacksdk-0.17.2-1.el7.noarch | 14:21 |
| pabelanger | nhicher: if that in logs, or using the clouds credentials? | 14:21 |
| pabelanger | nhicher: which version of keystoneauth is installed? | 14:23 |
| nhicher | using python3 zuul_swift_upload.py vexxhost-ansible-networking logs file | 14:23 |
| nhicher | rh-python35-python-keystoneauth1-3.8.0-1.el7.noarch | 14:23 |
| sfbender | Javier Peña created DLRN master: [Do not merge] Test https://softwarefactory-project.io/r/13714 | 14:23 |
| pabelanger | nhicher: are you able to debug more to why there is an auth issue? I can't figure it out from here. | 14:25 |
| nhicher | yes, I will install openstackclient to validate I can reach vexxhost from executor | 14:26 |
| pabelanger | thanks, I not longer have access to SF services, otherwise, I'd help | 14:28 |
| nhicher | that's strange it failed with os client too | 14:45 |
| nhicher | The request you have made requires authentication | 14:45 |
| nhicher | pabelanger: do you have to project_id for vexxhost ? | 14:51 |
| nhicher | I tried on ze02 with the credential mnaser provides for tripleo-ci jobs and it works | 14:51 |
| nhicher | the only difference is I have project_id on my cloud definition | 14:52 |
| pabelanger | nhicher: e2ec7f23-4152-44c8-9c90-309191f8f3fd | 14:54 |
| pabelanger | should be project | 14:54 |
| *** sshnaidm|pto has joined #softwarefactory | 14:56 | |
| nhicher | so using a *good* clouds.yaml and python3 zuul_swift_upload.py vexxhost-ansible-networking logs test_zuul_swift_upload.py => https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs | 15:00 |
| nhicher | pabelanger: ^^ | 15:00 |
| pabelanger | nhicher: what did you change? | 15:00 |
| pabelanger | nhicher: when we updated openstacksdk on executors last night, did we restart zuul? | 15:01 |
| nhicher | previously I got clouds.yaml from managesf, but I was outdate, the right one is on zuul-scheduler node | 15:01 |
| nhicher | *outdated | 15:01 |
| pabelanger | nhicher: is it the same as https://github.com/ansible-network/zuul-config/blob/master/zuul.d/secrets.yaml#L4 | 15:02 |
| *** sshnaidm|pto has quit IRC | 15:02 | |
| nhicher | https://softwarefactory-project.io/paste/show/1214/ | 15:06 |
| pabelanger | nhicher: I think we might have something wrong with the password, I am going to update zuul_encrypt and try again | 15:07 |
| nhicher | ok | 15:07 |
| pabelanger | nhicher: if that fails, we'll have to use --keep for executor and manually inspect the decrypted secrets | 15:08 |
| *** zoli|wfh is now known as zoli|gone | 15:12 | |
| *** zoli|gone is now known as zoli | 15:12 | |
| pabelanger | nhicher: okay, it was password that was issue | 15:15 |
| pabelanger | needed to use --strip on the encrypt | 15:15 |
| nhicher | cool =) | 15:15 |
| pabelanger | nhicher: give me a sec to really confirm, but thanks again for your help! | 15:16 |
| nhicher | pabelanger: you're welcome =) | 15:21 |
| pabelanger | okay, job passed | 15:23 |
| pabelanger | woot | 15:23 |
| pabelanger | but, error in swift | 15:23 |
| pabelanger | https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs_45/45/8d13b58b7114c0ae2150a1fc49113516f0b2487e/check/tox-docs/html/ | 15:23 |
| *** jpena is now known as jpena|off | 16:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!