| *** slaweq has quit IRC | 00:07 | |
| *** jesusaur has quit IRC | 00:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add fetch-output and ensure-output-dirs tests https://review.openstack.org/628731 | 00:19 |
|---|---|---|
| *** adriant has quit IRC | 00:19 | |
| *** jesusaur has joined #openstack-infra | 00:24 | |
| *** slaweq has joined #openstack-infra | 00:26 | |
| *** hongbin has quit IRC | 00:29 | |
| *** slaweq has quit IRC | 00:37 | |
| *** slaweq has joined #openstack-infra | 00:54 | |
| *** Adri2000 has quit IRC | 01:04 | |
| *** slaweq has quit IRC | 01:07 | |
| *** Adri2000 has joined #openstack-infra | 01:08 | |
| *** slaweq has joined #openstack-infra | 01:12 | |
| *** slaweq has quit IRC | 01:24 | |
| *** dave-mccowan has joined #openstack-infra | 01:35 | |
| *** Adri2000 has quit IRC | 01:38 | |
| *** Adri2000 has joined #openstack-infra | 01:43 | |
| *** slaweq has joined #openstack-infra | 01:49 | |
| *** slaweq has quit IRC | 01:55 | |
| *** wolverineav has joined #openstack-infra | 02:02 | |
| *** dave-mccowan has quit IRC | 02:07 | |
| *** slaweq has joined #openstack-infra | 02:12 | |
| *** slaweq has quit IRC | 02:25 | |
| *** hwoarang has quit IRC | 02:40 | |
| *** hwoarang has joined #openstack-infra | 02:42 | |
| *** hwoarang has quit IRC | 02:56 | |
| *** hwoarang has joined #openstack-infra | 02:59 | |
| *** slaweq has joined #openstack-infra | 03:03 | |
| *** slaweq has quit IRC | 03:25 | |
| *** wolverineav has quit IRC | 03:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:00 | |
| *** wolverineav has joined #openstack-infra | 04:05 | |
| *** bhavikdbavishi has quit IRC | 04:45 | |
| *** pcrews has joined #openstack-infra | 04:46 | |
| *** hwoarang has quit IRC | 05:09 | |
| *** hwoarang has joined #openstack-infra | 05:10 | |
| *** wolverineav has quit IRC | 05:13 | |
| *** wolverineav has joined #openstack-infra | 05:14 | |
| *** wolverineav has quit IRC | 05:18 | |
| *** jamesmcarthur has joined #openstack-infra | 05:46 | |
| *** bhavikdbavishi has joined #openstack-infra | 06:20 | |
| *** bhavikdbavishi has quit IRC | 06:33 | |
| openstackgerrit | melissaml proposed openstack-infra/grafyaml master: Removed older version of python3.4 https://review.openstack.org/628751 | 06:35 |
| *** bhavikdbavishi has joined #openstack-infra | 06:35 | |
| *** psachin has joined #openstack-infra | 06:36 | |
| *** jamesmcarthur has quit IRC | 06:43 | |
| *** bhavikdbavishi has quit IRC | 06:46 | |
| openstackgerrit | melissaml proposed openstack-infra/bindep master: Removed older version of python3.4 https://review.openstack.org/628752 | 06:58 |
| *** jamesmcarthur has joined #openstack-infra | 07:03 | |
| *** jamesmcarthur has quit IRC | 07:09 | |
| openstackgerrit | melissaml proposed openstack-dev/pbr master: Removed older version of python3.4 https://review.openstack.org/628754 | 07:21 |
| *** agopi_ has joined #openstack-infra | 07:29 | |
| *** xek has joined #openstack-infra | 07:36 | |
| *** agopi_ has quit IRC | 07:52 | |
| *** slaweq has joined #openstack-infra | 08:17 | |
| *** bhavikdbavishi has joined #openstack-infra | 09:09 | |
| *** Miouge has left #openstack-infra | 09:41 | |
| *** ssbarnea has quit IRC | 09:49 | |
| *** Miouge has joined #openstack-infra | 09:49 | |
| *** ssbarnea has joined #openstack-infra | 09:55 | |
| *** bhavikdbavishi has quit IRC | 10:10 | |
| openstackgerrit | weizj proposed openstack/os-testr master: Update hacking version https://review.openstack.org/628608 | 10:18 |
| *** yamamoto has joined #openstack-infra | 10:28 | |
| *** yamamoto has quit IRC | 10:45 | |
| *** hwoarang_ has joined #openstack-infra | 10:46 | |
| *** psachin has quit IRC | 10:50 | |
| *** hwoarang has quit IRC | 10:50 | |
| openstackgerrit | weizj proposed openstack/os-testr master: Update hacking version https://review.openstack.org/628608 | 10:59 |
| *** pbourke has quit IRC | 11:05 | |
| *** pbourke has joined #openstack-infra | 11:07 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:20 | |
| *** yamamoto has joined #openstack-infra | 11:27 | |
| *** hwoarang_ has quit IRC | 11:33 | |
| *** hwoarang has joined #openstack-infra | 11:49 | |
| *** dpawlik has joined #openstack-infra | 12:02 | |
| *** tosky has joined #openstack-infra | 12:04 | |
| *** bhavikdbavishi has quit IRC | 12:07 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:14 | |
| *** bhavikdbavishi has quit IRC | 12:50 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:58 | |
| *** whoami-rajat has joined #openstack-infra | 13:07 | |
| *** dave-mccowan has joined #openstack-infra | 13:15 | |
| *** hwoarang_ has joined #openstack-infra | 13:33 | |
| *** niedbalski has quit IRC | 13:33 | |
| *** hwoarang has quit IRC | 13:34 | |
| *** hwoarang has joined #openstack-infra | 13:47 | |
| *** hwoarang_ has quit IRC | 13:48 | |
| *** dpawlik has quit IRC | 13:54 | |
| *** dpawlik has joined #openstack-infra | 13:55 | |
| *** dpawlik has quit IRC | 13:59 | |
| *** dpawlik has joined #openstack-infra | 14:11 | |
| *** dpawlik has quit IRC | 14:15 | |
| *** dkehn_ has joined #openstack-infra | 14:23 | |
| *** dkehn has quit IRC | 14:23 | |
| *** dkehn_ is now known as dkehn | 14:23 | |
| *** markvoelker has quit IRC | 14:24 | |
| *** hwoarang_ has joined #openstack-infra | 14:50 | |
| *** hwoarang has quit IRC | 14:53 | |
| *** hwoarang has joined #openstack-infra | 15:00 | |
| *** hwoarang_ has quit IRC | 15:01 | |
| *** dave-mccowan has quit IRC | 15:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: DNM Testing fetch-output https://review.openstack.org/628668 | 15:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add fetch-output and ensure-output-dirs tests https://review.openstack.org/628731 | 15:22 |
| *** agopi_ has joined #openstack-infra | 15:41 | |
| *** Adri2000 has quit IRC | 15:43 | |
| *** Adri2000 has joined #openstack-infra | 15:47 | |
| *** agopi_ has quit IRC | 15:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run fetch-output before removing ssh keys https://review.openstack.org/628780 | 16:47 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add fetch-output and ensure-output-dirs tests https://review.openstack.org/628731 | 16:50 |
| mordred | AJaeger: yes - definitely need that - still not sure why it's failing on checking the logs though | 16:51 |
| corvus | mordred: in the error for checking the logs it says" The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'work_dir'" | 16:52 |
| mordred | oh - it does? | 16:52 |
| mordred | I'm just stupid | 16:52 |
| corvus | http://logs.openstack.org/31/628731/5/check/openstack-infra-base-integration-centos-7/8403b08/ara-report/result/adf148dd-2d07-4048-8cc9-18915573d65d/ | 16:52 |
| corvus | mordred: another thing: the roles that are being tested there are in a different repo; that should be fine (and even the file matchers should be okay with that, if we run that job on zuul-jobs); i just wanted to make sure that's known | 16:53 |
| mordred | corvus: yah - the job is actually triggered by zuul-jobs ... it's just in openstack-zuul-jobs because it's construction is specific to our deployment | 16:54 |
| corvus | kk | 16:54 |
| mordred | corvus: it's a lovely construct spanning project-config, zuul-jobs and openstack-zuul-jobs - my brain has been hurting just a little bit - but also, it works! | 16:54 |
| AJaeger | mordred: 628731 has first passes now! | 16:55 |
| mordred | AJaeger: ZOMG! that's so exciting | 16:55 |
| AJaeger | yeah, 2 passes that failed previously, so let's hope the rest passes as well ;) | 16:56 |
| AJaeger | mordred: why the post fails on https://review.openstack.org/#/c/628668/ ? | 16:57 |
| mordred | AJaeger: those will be fixed by 628780 | 16:58 |
| mordred | AJaeger: as soon as it lands we can recheck it | 16:58 |
| mordred | AJaeger: corvus pointed out that we were running the fetch-output role after the role to remove the ssh keys | 16:58 |
| mordred | this led to errors on the executor like: 2019-01-06 15:21:20,193 DEBUG zuul.AnsibleJob: [build: a2e957388b6144a19e45477bff20b3ad] Ansible output: b'fatal: [ubuntu-xenial]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o Port=22 --out-format=<<CHANGED>>%i %n%L zuul@198.72.124.165:/home/zuul/zuul-output/logs/ | 16:59 |
| mordred | /var/lib/zuul/builds/a2e957388b6144a19e45477bff20b3ad/work/logs/", "msg": "Permission denied (publickey).\\r\\nrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]\\nrsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]\\n", "rc": 255}' | 16:59 |
| AJaeger | mordred: I see... | 16:59 |
| clarkb | People are around today? in that case anyone want to review some straightforward puppet futureparser group changes? https://review.openstack.org/#/c/616292/5 is the current bottom of the stack. I'm happy to approve them and babysit | 17:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Run fetch-output before removing ssh keys https://review.openstack.org/628780 | 17:00 |
| clarkb | mordred: ^ | 17:00 |
| mordred | clarkb: looking | 17:01 |
| AJaeger | mordred: what's wrong here: http://logs.openstack.org/31/628731/6/check/openstack-infra-multinode-integration-centos-7/dbb05e5/job-output.txt.gz#_2019-01-06_16_57_51_569690 ? | 17:01 |
| AJaeger | mordred: is that the same problem? | 17:02 |
| mordred | AJaeger: I donm't think so? | 17:03 |
| mordred | oh - hrm | 17:04 |
| mordred | ok. this will take a little more care ... I think this is a race condition which won't normally happen - the idea is that for multi-node jobs there should really only be artifacts or docs on one host - or that if there are multiples they shouldn't conflict | 17:05 |
| clarkb | each host gets its own dir so they shouldn't conflict right? | 17:06 |
| mordred | but maybe a safer solution would be to add a serial tag to the role so that it doesn't try to rsync the same file in parallel | 17:06 |
| mordred | clarkb: only for logs | 17:06 |
| clarkb | ah | 17:06 |
| mordred | because the idea is that host-specificness of logs is important, whereas docs and artifacts are things that are outputs of the job so which node they came from is irrelevant | 17:07 |
| clarkb | yup makes sense | 17:08 |
| clarkb | (probably worth another note inthe readme too?) | 17:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add fetch-output and ensure-output-dirs tests https://review.openstack.org/628731 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add note about multi-node jobs https://review.openstack.org/628781 | 17:14 |
| mordred | clarkb: how's what? ^^ | 17:14 |
| clarkb | +2 | 17:15 |
| mordred | clarkb: +2 for the stack | 17:16 |
| clarkb | mordred: thanks. I approved the first one and will keep an eye on ze01 after that goes in | 17:17 |
| *** martinkennelly has joined #openstack-infra | 17:20 | |
| martinkennelly | Ive tried configuring my gerrit trigger in jenkins with the the commands in the red circle but it silently fails. gerrit review works but not gerrit approve. https://imgur.com/a/aP8fA7E | 17:22 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for one zuul executor https://review.openstack.org/616292 | 17:24 |
| mordred | clarkb: just got an unrelated test fail related to ovs_ping testing - do you want to look at it to see if there's something real there? or just recheck past it? | 17:27 |
| mordred | clarkb: http://logs.openstack.org/31/628731/7/check/openstack-infra-multinode-integration-ubuntu-trusty/35c4982/ if you want to look | 17:27 |
| *** agopi_ has joined #openstack-infra | 17:29 | |
| clarkb | hrm | 17:30 |
| clarkb | http://logs.openstack.org/31/628731/7/check/openstack-infra-multinode-integration-ubuntu-trusty/35c4982/job-output.txt.gz#_2019-01-06_17_19_06_990649 shows us the ports on either side | 17:30 |
| clarkb | mordred: I think if a recheck has it working then maybe this is another version of the subnode communication failure over ipv4 that tripleo has on limestone | 17:32 |
| *** e0ne has joined #openstack-infra | 17:32 | |
| clarkb | mordred: the ovs vxlan tunnel is set up over ipv4 on the private network in that cloud, but the zuul ansible ssh connection is over ipv6 and works | 17:32 |
| clarkb | we have the hypervisor id info now too so maybe that can help logan- tell us if it is a cloud side issue or not | 17:32 |
| mordred | clarkb: nod | 17:34 |
| *** e0ne has quit IRC | 17:36 | |
| *** tosky has quit IRC | 17:51 | |
| mordred | AJaeger, corvus, clarkb: https://review.openstack.org/#/c/628668/ passed! yay! | 18:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add fetch-output to base job https://review.openstack.org/511851 | 18:07 |
| mordred | that should be good to go now ^^ | 18:07 |
| clarkb | mordred: where does the post-ssh chagne come in? | 18:15 |
| mordred | clarkb: what do you mean? | 18:16 |
| mordred | clarkb: oh- why are we changing post-ssh to just post? | 18:16 |
| clarkb | ya | 18:16 |
| mordred | https://review.openstack.org/#/c/628780/ is the base-test patch where we did it - but it's because running fetch-output in post-logs doesn't work because post-ssh removes the ssh keys | 18:17 |
| mordred | so we need to put fetch-output in post-ssh - which then makes it more general and not specific to ssh tasks | 18:17 |
| mordred | I guess I could put that in the actual commit message | 18:17 |
| clarkb | oh its a rename | 18:17 |
| clarkb | ya I see that now | 18:17 |
| mordred | yeah | 18:18 |
| mordred | want me to update commit message? | 18:18 |
| clarkb | I'm ok with it after realize that rename fact | 18:18 |
| mordred | kk. cool | 18:18 |
| clarkb | whcih gerrit does try to show me | 18:18 |
| AJaeger | mordred: Yeah! Party time ;) | 18:39 |
| AJaeger | clarkb: could you review https://review.openstack.org/628674 as well, please? I'd like to see tests merged first. | 18:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Add note about multi-node jobs https://review.openstack.org/628781 | 18:41 |
| AJaeger | mordred: please approve https://review.openstack.org/#/c/511851/ when you are around to monitor... | 18:44 |
| *** whoami-rajat has quit IRC | 19:06 | |
| *** hongbin has joined #openstack-infra | 19:08 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add note about multi-node jobs https://review.openstack.org/628781 | 19:17 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for all zuul executors https://review.openstack.org/616293 | 19:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for one zuul merger https://review.openstack.org/616294 | 19:21 |
| *** armax has joined #openstack-infra | 19:39 | |
| *** e0ne has joined #openstack-infra | 20:00 | |
| *** agopi_ has quit IRC | 20:07 | |
| *** e0ne has quit IRC | 20:12 | |
| *** bhavikdbavishi has quit IRC | 20:14 | |
| *** rkukura has quit IRC | 20:30 | |
| *** chason has quit IRC | 20:30 | |
| *** xek has quit IRC | 20:37 | |
| *** adriant has joined #openstack-infra | 21:19 | |
| *** chason has joined #openstack-infra | 21:21 | |
| *** chason has quit IRC | 21:25 | |
| *** chason has joined #openstack-infra | 21:27 | |
| prometheanfire | does the bot have a !seen function for when/where it last saw a nick? | 21:31 |
| fungi | mordred: clarkb: is there a good way to explain how logs differ from any other job artifacts? it seems to me like it would be simpler to treat all job artifacts with similar host-specific precaution | 22:13 |
| clarkb | fungi logs arehost specific artifacts and "artifacts" arejob specific might be one way? | 22:16 |
| clarkb | every hosthas a syslog for example | 22:17 |
| clarkb | but you only have one copy of that dockerimage | 22:17 |
| mordred | yeah. that's a good example | 22:28 |
| mordred | in general - I think it's doubtful that many multi-node jobs will be producing docs or artifacts | 22:28 |
| mordred | and if they did - say like if you spun up 4 nodes to do something k8s-y - it's likely that the artifacts are actually only going to be on one of the nodes | 22:29 |
| *** hongbin has quit IRC | 23:02 | |
| *** rcernin has joined #openstack-infra | 23:06 | |
| fungi | okay, just trying to reconcile the complexity of treating them differently (and so having more than one kind of thing) | 23:10 |
| *** yamamoto has joined #openstack-infra | 23:18 | |
| *** yamamoto has quit IRC | 23:22 | |
| *** bobh has joined #openstack-infra | 23:24 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!