| clarkb | pabelanger: it might help to find the error on the executor side too | 00:00 |
|---|---|---|
| clarkb | just to confirm which exact code path did it | 00:00 |
| pabelanger | clarkb: no much help sadly, 1 sec | 00:00 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change of day and channel for bi-weekly meetings for Public Cloud WG. https://review.openstack.org/553022 | 00:01 |
| ianw | persia: do you want me to provide a patch? i forget exactly but there's a couple of tweaks to the debian/* files | 00:01 |
| pabelanger | clarkb: http://paste.openstack.org/show/701006/ | 00:01 |
| pabelanger | clarkb: I'm going to see if I can write a quick unit test to expose the failure | 00:01 |
| clarkb | pabelanger: _is_safe_path is my guess | 00:02 |
| clarkb | which you should be able to execute on its own just passing in a little data | 00:03 |
| clarkb | one thing that it could be is a symlink in the repo | 00:03 |
| *** dingyichen has joined #openstack-infra | 00:03 | |
| pabelanger | k | 00:04 |
| pabelanger | not sure if we use a symlink for src_root | 00:04 |
| *** dingyichen has quit IRC | 00:04 | |
| clarkb | shouldn't be for src_root, but if there is a symlink in the repo itself trying to get out that should be caught by this | 00:04 |
| pabelanger | oh, maybe | 00:05 |
| clarkb | pabelanger: what repo is it trying to sync? | 00:05 |
| clarkb | or I guess it could be more than one repo | 00:05 |
| pabelanger | clarkb: nodepool, zuul, diskimage-builder, gear: http://logs.openstack.org/00/553100/1/check/windmill-ubuntu-xenial/4d1b0a0/zuul-info/inventory.yaml | 00:05 |
| pabelanger | shade | 00:05 |
| *** olaph1 has joined #openstack-infra | 00:06 | |
| *** dizquierdo has joined #openstack-infra | 00:06 | |
| clarkb | so if any of them have a symlink out of the repo this would trap on that | 00:06 |
| *** yamamoto has joined #openstack-infra | 00:07 | |
| clarkb | oh wait | 00:07 |
| clarkb | zuul itself has this problem right? | 00:07 |
| clarkb | bwahahaha | 00:07 |
| *** olaph has quit IRC | 00:07 | |
| clarkb | this will be pretty epic if the cause | 00:07 |
| pabelanger | don't understand question | 00:08 |
| *** dingyichen has joined #openstack-infra | 00:08 | |
| clarkb | pabelanger: as part of fixing this bug tests were added with real symlinks to things like /opt/assemble | 00:08 |
| clarkb | those symlinks are now breaking your tests because they would copy inappropriate things from the test node | 00:08 |
| clarkb | corvus: tobiash ^ | 00:09 |
| pabelanger | Oh, haha, nice | 00:09 |
| pabelanger | I can test it quickly | 00:09 |
| clarkb | if this is the case then I think the code fix is working properly | 00:09 |
| clarkb | we've just wedged ourselves in a fun way by having tests that are too good :P | 00:09 |
| corvus | clarkb: iiuc, there is not a systemic problem with zuul, just a problem with windmill doing things with zuul's wacky test symlinks? | 00:10 |
| corvus | ie, this is not affecting anything that isn't actually operating on the zuul source code repo itself, right? | 00:10 |
| clarkb | corvus: ya, though its possible we would have problems with syncing zuul repo for zuul jobs now? | 00:10 |
| corvus | clarkb: we git push, so i don't think it'd be a prob | 00:11 |
| pabelanger | well, we no longer use syncronize of repos from executor to nodes, we git clone --mirror them | 00:11 |
| clarkb | oh yes correct | 00:11 |
| *** katkapilatova has quit IRC | 00:11 | |
| pabelanger | but, that can't be run in an untrusted context | 00:11 |
| clarkb | so ya this would be windmills weirdness and I think zuul is doing the correct thing here | 00:11 |
| clarkb | pabelanger: couldn't you make zuul a required project for windmill jobs? | 00:11 |
| corvus | this doesn't seem critical, so i'm going to eod now; happy to help tomorrow | 00:11 |
| clarkb | ya I think the scope of this is small | 00:12 |
| clarkb | and the good news is the bug fix seems to be working :P | 00:12 |
| pabelanger | clarkb: I do, but the use case is I'm only copying git repos to a single node (in multi node job). Today our base jobs mirror to all nodes | 00:13 |
| clarkb | pabelanger: that is an optimization though right? | 00:13 |
| pabelanger | there is a way we can fix it with negative host filters, but haven't worked on that yet | 00:13 |
| clarkb | like it won't really hurt anything to have zuul on more than one node | 00:13 |
| pabelanger | clarkb: well, in the case of my inventory file, I can't actually use a base job because of race conditions | 00:14 |
| pabelanger | Usually serial does work, but we need to patch zuul to support extra-vars | 00:14 |
| pabelanger | clarkb: the issue is, this nodeset: http://git.openstack.org/cgit/openstack/windmill/tree/.zuul.d/jobs.yaml#n62 | 00:15 |
| pabelanger | to ansible is it multiple hosts, but really a single node from nodepool | 00:15 |
| pabelanger | and our base playbooks / roles setup to work that way. And will fail due to race conditions between the hosts | 00:16 |
| *** rlandy has quit IRC | 00:16 | |
| pabelanger | so, I've been able to create a base playbook, atop of base-minimal which was working until today: http://git.openstack.org/cgit/openstack/windmill/tree/tests/base-minimal.yaml | 00:17 |
| pabelanger | I think if our playbook / role were more idempotent it might also help | 00:17 |
| pabelanger | but, would need to be tested more | 00:18 |
| pabelanger | clarkb: I think your logic is correct, dropping zuul from synchronize gets job passed the failure | 00:19 |
| *** eernst has quit IRC | 00:20 | |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/prepare-workspace/tasks/main.yaml would also be broken for zuul users in untrusted context (required-project of zuul) | 00:20 |
| *** eernst has joined #openstack-infra | 00:21 | |
| *** xinliang has quit IRC | 00:22 | |
| clarkb | pabelanger: ya we could have the update role check if the repos already exist and what their git HEAD is | 00:22 |
| clarkb | (to avoid resyncing if not necessary) | 00:22 |
| *** dizquierdo has quit IRC | 00:25 | |
| pabelanger | clarkb: as for zuul and /opt symlinks, could those be create a job runtime and avoid the need from sourcing them intree? | 00:26 |
| clarkb | pabelanger: probably, thats something to talk to tobias about | 00:26 |
| pabelanger | wfm | 00:27 |
| pabelanger | for now, I can work around the failure | 00:27 |
| *** diablo_rojo has quit IRC | 00:28 | |
| *** odyssey4me has quit IRC | 00:29 | |
| *** odyssey4me has joined #openstack-infra | 00:29 | |
| *** eernst has quit IRC | 00:29 | |
| *** diablo_rojo has joined #openstack-infra | 00:31 | |
| *** xinliang has joined #openstack-infra | 00:34 | |
| pabelanger | clarkb: http://git.openstack.org/cgit/openstack-infra/zuul/tree/tests/fixtures/config/remote-action-modules/git/org_project/playbooks/roles/copy-test/files/symlink will be the issue | 00:34 |
| pabelanger | so yay, out safe_path logic is working :) | 00:34 |
| clarkb | pabelanger: ya there is more than one of them too | 00:34 |
| pabelanger | yah | 00:35 |
| pabelanger | so, should be able to create them at runtime, but will have tobiash first confirm | 00:35 |
| clarkb | also we may need that idempotency for status nodes anyways | 00:39 |
| clarkb | so probably a good thing to fix | 00:39 |
| clarkb | *static | 00:39 |
| *** HeOS has quit IRC | 00:40 | |
| *** s-shiono has joined #openstack-infra | 00:41 | |
| *** stakeda has joined #openstack-infra | 00:44 | |
| *** wolverin_ has quit IRC | 00:46 | |
| *** wolverineav has joined #openstack-infra | 00:46 | |
| ianw | anyone know off hand what copies in /etc/puppet/hieradata/production ? i'm having trouble launching mirror-update01.o.o | 00:46 |
| *** diablo_rojo has quit IRC | 00:47 | |
| pabelanger | ansible-role-puppet I think | 00:47 |
| ianw | oh, i know what it is | 00:48 |
| *** r-daneel has quit IRC | 00:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Match numbered mirror-update servers https://review.openstack.org/553111 | 00:49 |
| ianw | pabelanger: ^ that, if you don't mind a review :) | 00:49 |
| *** wolverineav has quit IRC | 00:51 | |
| pabelanger | +2 | 00:51 |
| ianw | does it need a ~ before it? what does that mena | 00:52 |
| *** diablo_rojo has joined #openstack-infra | 00:53 | |
| pabelanger | oh, maybe? I'm not sure what it does. I've added it because of copypasta | 00:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Match numbered mirror-update servers https://review.openstack.org/553111 | 00:53 |
| ianw | heh, well now that has it too :0 | 00:53 |
| *** dprince has quit IRC | 00:58 | |
| *** dave-mccowan has joined #openstack-infra | 01:05 | |
| persia | ianw: Sorry: was called away. No worries, I'm fine playing with debdiff to see what you've done. | 01:07 |
| *** bobh has joined #openstack-infra | 01:10 | |
| *** bobh has quit IRC | 01:11 | |
| openstackgerrit | megan guiney proposed openstack-infra/subunit2sql master: Add ability to grab stream processing variables from kwargs https://review.openstack.org/543836 | 01:14 |
| *** mriedem has quit IRC | 01:24 | |
| *** nicolasbock has quit IRC | 01:29 | |
| *** cshastri has joined #openstack-infra | 01:30 | |
| *** andreas_s has joined #openstack-infra | 01:33 | |
| *** r-daneel has joined #openstack-infra | 01:37 | |
| *** andreas_s has quit IRC | 01:37 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/550978 | 01:39 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects routes https://review.openstack.org/550979 | 01:39 |
| *** cshastri has quit IRC | 01:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 01:39 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Rename javascript package to zuul-dashboard https://review.openstack.org/551999 | 01:40 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 01:40 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 01:40 |
| *** cshastri has joined #openstack-infra | 01:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 01:41 |
| *** nicolasbock has joined #openstack-infra | 01:41 | |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Use -python35-jobs-horizon for trove-dashboard https://review.openstack.org/553116 | 01:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Shift html templates into components https://review.openstack.org/551327 | 01:43 |
| *** yamahata has quit IRC | 01:43 | |
| *** iyamahat has quit IRC | 01:46 | |
| openstackgerrit | Merged openstack-infra/system-config master: Match numbered mirror-update servers https://review.openstack.org/553111 | 01:46 |
| *** hongbin has joined #openstack-infra | 01:46 | |
| *** claudiub|2 has quit IRC | 01:55 | |
| *** harlowja has quit IRC | 01:59 | |
| *** salv-orl_ has joined #openstack-infra | 02:04 | |
| *** rosmaita has quit IRC | 02:04 | |
| *** namnh has joined #openstack-infra | 02:05 | |
| *** salv-orlando has quit IRC | 02:07 | |
| *** r-daneel has quit IRC | 02:15 | |
| *** r-daneel has joined #openstack-infra | 02:16 | |
| *** luzC has joined #openstack-infra | 02:18 | |
| *** david-lyle has quit IRC | 02:21 | |
| dmsimard | ianw: btw I replied on https://review.openstack.org/#/c/552707/ | 02:28 |
| *** zhurong has joined #openstack-infra | 02:29 | |
| ianw | dmsimard: why not just update it all? i don't care that much though | 02:30 |
| *** xarses_ has joined #openstack-infra | 02:30 | |
| *** xarses has quit IRC | 02:31 | |
| dmsimard | ianw: I followed your linaro patches fwiw :p | 02:32 |
| dmsimard | ianw: I just figured I wouldn't touch nodepool until the mirror was ready in case I screw up and nodepool starts building images or firing images :p | 02:33 |
| ianw | yeah, and i forgot it in my patches, which is why i called it out :) | 02:34 |
| ianw | that won't happen until there's config, which i agree comes after | 02:34 |
| dmsimard | I tagged my patches with the same topic on purpose, I'll try and document the process a little bit | 02:38 |
| openstackgerrit | Merged openstack-infra/grafyaml master: Support formatting. https://review.openstack.org/550045 | 02:46 |
| *** psachin has joined #openstack-infra | 02:46 | |
| *** wolverineav has joined #openstack-infra | 02:49 | |
| ianw | #status log mirror-update.o.o upgraded to xenial host mirror-update01.o.o. mirror-update turned into a cname for 01. old server remains at mirror-update-old.o.o but turned off, so as not run conflicting jobs; will clean up later | 02:54 |
| openstackstatus | ianw: finished logging | 02:54 |
| openstackgerrit | Merged openstack-infra/grafyaml master: Add gauge caps for singlestat panel. https://review.openstack.org/550474 | 02:56 |
| ianw | urgh, our puppet might not be as good at importing pgp keys as it should be ... | 03:00 |
| *** olaph1 has quit IRC | 03:06 | |
| *** olaph has joined #openstack-infra | 03:07 | |
| *** wolverineav has quit IRC | 03:10 | |
| *** eernst has joined #openstack-infra | 03:10 | |
| *** sree has joined #openstack-infra | 03:10 | |
| *** wolverineav has joined #openstack-infra | 03:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add debian security gpg key https://review.openstack.org/553125 | 03:11 |
| *** dave-mccowan has quit IRC | 03:14 | |
| *** germs has quit IRC | 03:14 | |
| *** wolverineav has quit IRC | 03:15 | |
| *** germs has joined #openstack-infra | 03:15 | |
| *** germs has quit IRC | 03:15 | |
| *** germs has joined #openstack-infra | 03:15 | |
| *** germs has quit IRC | 03:16 | |
| *** germs has joined #openstack-infra | 03:17 | |
| *** germs has quit IRC | 03:17 | |
| *** germs has joined #openstack-infra | 03:17 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 03:18 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 03:18 |
| *** annp has joined #openstack-infra | 03:26 | |
| *** nicolasbock has quit IRC | 03:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add debian security gpg key https://review.openstack.org/553125 | 03:30 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add second Ubuntu signing key https://review.openstack.org/553130 | 03:30 |
| *** felipemonteiro_ has joined #openstack-infra | 03:32 | |
| *** tojuvone_ is now known as tojuvone | 03:32 | |
| *** chrisyang_0660 has joined #openstack-infra | 03:33 | |
| *** felipemonteiro__ has joined #openstack-infra | 03:33 | |
| *** janki has joined #openstack-infra | 03:34 | |
| *** ramishra has quit IRC | 03:35 | |
| *** ramishra has joined #openstack-infra | 03:35 | |
| *** felipemonteiro_ has quit IRC | 03:36 | |
| *** links has joined #openstack-infra | 03:41 | |
| *** felipemonteiro__ has quit IRC | 03:45 | |
| *** felipemonteiro__ has joined #openstack-infra | 03:46 | |
| *** ramishra has quit IRC | 03:49 | |
| *** ramishra has joined #openstack-infra | 03:50 | |
| *** hongbin has quit IRC | 03:52 | |
| *** zhurong has quit IRC | 03:53 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add debian security gpg key https://review.openstack.org/553125 | 04:07 |
| *** germs has quit IRC | 04:09 | |
| *** harlowja has joined #openstack-infra | 04:11 | |
| *** Swami has quit IRC | 04:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 04:21 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Fix debian key typo https://review.openstack.org/553137 | 04:26 |
| *** eernst has quit IRC | 04:31 | |
| *** andreas_s has joined #openstack-infra | 04:31 | |
| *** iyamahat_ has joined #openstack-infra | 04:33 | |
| *** dhajare has joined #openstack-infra | 04:34 | |
| *** gyee has quit IRC | 04:35 | |
| *** andreas_s has quit IRC | 04:36 | |
| *** jamesmcarthur has quit IRC | 04:41 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add second Ubuntu signing key https://review.openstack.org/553130 | 04:43 |
| *** lpetrut has joined #openstack-infra | 04:46 | |
| *** harlowja has quit IRC | 04:50 | |
| *** VW has joined #openstack-infra | 04:50 | |
| *** yamahata has joined #openstack-infra | 04:50 | |
| *** vivsoni has joined #openstack-infra | 05:00 | |
| *** felipemonteiro__ has quit IRC | 05:01 | |
| *** VW has quit IRC | 05:06 | |
| *** VW has joined #openstack-infra | 05:07 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix debian key typo https://review.openstack.org/553137 | 05:07 |
| *** vivsoni has quit IRC | 05:08 | |
| *** ykarel|afk has joined #openstack-infra | 05:10 | |
| *** ykarel|afk is now known as ykarel | 05:10 | |
| *** pgadiya has joined #openstack-infra | 05:11 | |
| *** lpetrut has quit IRC | 05:11 | |
| *** alexchadin has joined #openstack-infra | 05:11 | |
| *** VW has quit IRC | 05:11 | |
| *** _jbadiapa has quit IRC | 05:12 | |
| *** udesale has joined #openstack-infra | 05:13 | |
| *** rossella_s has quit IRC | 05:14 | |
| *** alexchadin has quit IRC | 05:17 | |
| *** rossella_s has joined #openstack-infra | 05:17 | |
| *** alexchadin has joined #openstack-infra | 05:17 | |
| *** agopi has joined #openstack-infra | 05:22 | |
| adriant | is something wrong with gerrit? | 05:25 |
| adriant | I can't do "git review" for my patch right now | 05:25 |
| adriant | It works for our internal company gerrit, but not the openstack one. | 05:26 |
| adriant | it's not even giving me errors, just doing nothing. | 05:26 |
| *** agopi has quit IRC | 05:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: model: freeze parent playbooks https://review.openstack.org/553144 | 05:37 |
| ianw | adriant: not that i know of ... are the packets getting out of your company? | 05:44 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Revert "Don't store references to secret objects from jobs" https://review.openstack.org/553147 | 05:45 |
| *** claudiub|2 has joined #openstack-infra | 05:48 | |
| *** madhuvishy has left #openstack-infra | 05:53 | |
| *** rossella_s has quit IRC | 06:04 | |
| *** rossella_s has joined #openstack-infra | 06:05 | |
| *** rossella_s has quit IRC | 06:10 | |
| *** rossella_s has joined #openstack-infra | 06:11 | |
| *** jbadiapa has joined #openstack-infra | 06:18 | |
| *** iyamahat_ has quit IRC | 06:28 | |
| *** yamahata has quit IRC | 06:28 | |
| *** dbecker has quit IRC | 06:32 | |
| *** aeng has quit IRC | 06:34 | |
| *** olaph1 has joined #openstack-infra | 06:34 | |
| *** olaph has quit IRC | 06:35 | |
| *** rossella_s has quit IRC | 06:36 | |
| *** rossella_s has joined #openstack-infra | 06:36 | |
| *** dhajare has quit IRC | 06:39 | |
| *** rossella_s has quit IRC | 06:42 | |
| *** rossella_s has joined #openstack-infra | 06:42 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/551447 | 06:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/551448 | 06:43 |
| *** dbecker has joined #openstack-infra | 06:46 | |
| *** jcoufal has joined #openstack-infra | 06:51 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/552316 | 06:52 |
| *** namnh has quit IRC | 06:53 | |
| *** aviau has quit IRC | 07:11 | |
| *** jbadiapa has quit IRC | 07:12 | |
| *** aviau has joined #openstack-infra | 07:14 | |
| *** alexchadin has quit IRC | 07:14 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** alexchadin has joined #openstack-infra | 07:19 | |
| *** dsariel has joined #openstack-infra | 07:21 | |
| *** gridinv has quit IRC | 07:21 | |
| frickler | infra-root: I'm seeing strange POST_FAILUREs that look like they might be related to recent zuul patches, e.g. http://logs.openstack.org/86/547986/7/check/tempest-full-py36/6e6bb4f/job-output.txt.gz#_2018-03-15_03_03_39_413259 | 07:25 |
| AJaeger | frickler: related to https://review.openstack.org/553147 ? | 07:25 |
| frickler | AJaeger: hmm, not sure. lots of others are simple cut off while uploading logs http://logs.openstack.org/86/547986/7/check/tempest-full/1338109/job-output.txt.gz#_2018-03-15_04_30_32_432979 | 07:27 |
| *** andreas_s has joined #openstack-infra | 07:27 | |
| *** dhajare has joined #openstack-infra | 07:29 | |
| AJaeger | frickler: just a shot in the dark - no time for investigation | 07:30 |
| *** kjackal has quit IRC | 07:36 | |
| *** apetrich has quit IRC | 07:36 | |
| adriant | ianw: worked now. I'll blame solar flares. | 07:37 |
| *** armaan has joined #openstack-infra | 07:41 | |
| *** alexchadin has quit IRC | 07:41 | |
| *** alexchadin has joined #openstack-infra | 07:42 | |
| frickler | seems all tempest-* jobs are affected starting around 20:00 yesterday | 07:43 |
| AJaeger | tobiash: any ideas? ^ | 07:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/533993 | 07:45 |
| tobiash | frickler, AJaeger: that might be caused by the no_log fix | 07:46 |
| tobiash | looks like it strips a bit too much | 07:46 |
| tobiash | do you need no_log on this task? | 07:46 |
| tobiash | frickler, AJaeger: it seems that no_log now strips the stat from the result variable of http://logs.openstack.org/86/547986/7/check/tempest-full-py36/6e6bb4f/job-output.txt.gz#_2018-03-15_03_03_36_624026 | 07:48 |
| *** apetrich has joined #openstack-infra | 07:48 | |
| *** pcaruana has joined #openstack-infra | 07:48 | |
| *** lpetrut has joined #openstack-infra | 07:49 | |
| *** pcaruana has quit IRC | 07:50 | |
| *** pcaruana has joined #openstack-infra | 07:50 | |
| frickler | tobiash: so we should remove the no_log as a workaround? | 07:50 |
| *** pcaruana is now known as pcaruana|wfh| | 07:53 | |
| *** AJaeger has quit IRC | 07:56 | |
| *** hashar has joined #openstack-infra | 07:56 | |
| *** e0ne has joined #openstack-infra | 07:57 | |
| *** ykarel is now known as ykarel|lunch | 07:57 | |
| *** kjackal has joined #openstack-infra | 08:01 | |
| *** olaph has joined #openstack-infra | 08:03 | |
| *** olaph1 has quit IRC | 08:04 | |
| *** dingyichen has quit IRC | 08:06 | |
| *** HeOS has joined #openstack-infra | 08:08 | |
| *** witek has quit IRC | 08:11 | |
| *** witek_ has joined #openstack-infra | 08:11 | |
| tobiash | frickler: if this task doesn;t deal with secrets, yes remove it | 08:12 |
| *** zoli is now known as zoli|gone | 08:17 | |
| *** zoli|gone is now known as zoli | 08:17 | |
| *** AJaeger has joined #openstack-infra | 08:20 | |
| *** ramishra has quit IRC | 08:20 | |
| *** ramishra has joined #openstack-infra | 08:21 | |
| *** tesseract has joined #openstack-infra | 08:24 | |
| *** tmorin has joined #openstack-infra | 08:25 | |
| *** bhavik1 has joined #openstack-infra | 08:26 | |
| *** ykarel|lunch is now known as ykarel | 08:26 | |
| *** bhavik1 has quit IRC | 08:29 | |
| *** _alastor_ has quit IRC | 08:35 | |
| *** tosky has joined #openstack-infra | 08:40 | |
| *** jpena|off is now known as jpena | 08:43 | |
| tobiash | frickler: after thinking about that combining register with no_log in ansible feels kinda scary to me | 08:47 |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| *** rcernin has quit IRC | 09:00 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| *** ispp is now known as burned | 09:04 | |
| *** rossella_s has quit IRC | 09:05 | |
| *** rossella_s has joined #openstack-infra | 09:06 | |
| *** r-daneel has quit IRC | 09:07 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/543403 | 09:11 |
| *** lucas-afk is now known as lucasagomes | 09:12 | |
| *** alexchadin has quit IRC | 09:12 | |
| *** alexchadin has joined #openstack-infra | 09:13 | |
| *** dbecker_ has joined #openstack-infra | 09:13 | |
| *** alexchadin has quit IRC | 09:13 | |
| *** dbecker_ has left #openstack-infra | 09:14 | |
| *** dbecker has quit IRC | 09:14 | |
| *** alexchadin has joined #openstack-infra | 09:14 | |
| *** dbecker has joined #openstack-infra | 09:14 | |
| *** alexchadin has quit IRC | 09:14 | |
| *** alexchadin has joined #openstack-infra | 09:15 | |
| *** alexchadin has quit IRC | 09:15 | |
| *** alexchadin has joined #openstack-infra | 09:15 | |
| *** alexchadin has quit IRC | 09:16 | |
| *** zoli is now known as zoli|wfh | 09:16 | |
| *** zoli|wfh is now known as zoli | 09:16 | |
| *** alexchadin has joined #openstack-infra | 09:16 | |
| *** alexchadin has quit IRC | 09:16 | |
| *** dbecker has quit IRC | 09:22 | |
| *** dbecker has joined #openstack-infra | 09:22 | |
| *** shardy has joined #openstack-infra | 09:25 | |
| *** nicolasbock has joined #openstack-infra | 09:27 | |
| *** janki has quit IRC | 09:28 | |
| *** jbadiapa has joined #openstack-infra | 09:30 | |
| *** claudiub|2 is now known as claudiub | 09:32 | |
| *** alexchadin has joined #openstack-infra | 09:32 | |
| *** witek_ has quit IRC | 09:33 | |
| *** amoralej|off is now known as amoralej | 09:34 | |
| *** witek has joined #openstack-infra | 09:38 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| *** efoley has joined #openstack-infra | 09:40 | |
| alexchadin | hi. I'd like to put to job-output only logs about tempest run. for example, from this point http://logs.openstack.org/90/551790/24/check/watcher-tempest-zuulv3/a0143d5/job-output.txt.gz#_2018-03-15_09_04_25_620834 | 09:43 |
| *** dizquierdo has joined #openstack-infra | 09:44 | |
| alexchadin | do I need some post playbook? | 09:44 |
| *** florianf has joined #openstack-infra | 09:50 | |
| *** s-shiono has quit IRC | 09:53 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 09:54 |
| tosky | uhm, did anyone notice POST_FAILURES in some (devstack-based) jobs? https://review.openstack.org/#/c/553268/ | 09:54 |
| frickler | tosky: yes, see my discussion with tobiash earlier. reason not clear yet, we will probably need to wait for some US people to look into it | 09:56 |
| tosky | ups, I was not connected, thanks, I will recheck the logs | 09:57 |
| tobiash | frickler: did you try the workaround? | 09:57 |
| frickler | tobiash: no, as I'm not certain whether there might be secrets involved | 09:58 |
| alexchadin | tosky: +1, the same problem | 09:58 |
| dulek | frickler: Had you noticed that it fails on "apache-logs-conf : Create hard links"? | 10:02 |
| dulek | frickler: And seems like in logs for this task we only have "Output suppressed because no_log was given" | 10:03 |
| frickler | dulek: ansible seems to fail in various tasks with the log simply ending mid-task. the common theme I'm seeing is that they are executed directly after a task that has "no_log: True" | 10:05 |
| *** electrofelix has joined #openstack-infra | 10:05 | |
| dulek | frickler: Okay. So I guess it's another day with broken gates for Kuryr. ;) | 10:06 |
| *** stakeda has quit IRC | 10:10 | |
| *** pbourke has joined #openstack-infra | 10:11 | |
| *** rossella_s has quit IRC | 10:12 | |
| *** sree has quit IRC | 10:13 | |
| *** sree has joined #openstack-infra | 10:14 | |
| *** rossella_s has joined #openstack-infra | 10:14 | |
| *** shardy has quit IRC | 10:18 | |
| *** sree has quit IRC | 10:19 | |
| *** shardy has joined #openstack-infra | 10:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Generate symlinks during tests https://review.openstack.org/553316 | 10:21 |
| frickler | infra-root: who wants to look after backup jobs? cron job on zuul01.o.o produces "/bin/sh: 1: bup: not found" | 10:23 |
| dmellado | tosky: yeah | 10:26 |
| dmellado | seems that is affecting all jobs | 10:26 |
| dmellado | another tough day in the infra world | 10:26 |
| *** rossella_s has quit IRC | 10:28 | |
| *** florianf has quit IRC | 10:30 | |
| *** pblaho has joined #openstack-infra | 10:31 | |
| *** cshastri has quit IRC | 10:31 | |
| *** rossella_s has joined #openstack-infra | 10:31 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 10:35 |
| *** rossella_s has quit IRC | 10:38 | |
| *** rossella_s has joined #openstack-infra | 10:40 | |
| *** olaph1 has joined #openstack-infra | 10:45 | |
| *** olaph has quit IRC | 10:46 | |
| *** alexchadin has quit IRC | 10:46 | |
| tobiash | frickler: if that's blocking too much you also could do a local revert of the executors to the change before the no_log fix | 10:51 |
| *** erlon has joined #openstack-infra | 10:54 | |
| *** gongysh has joined #openstack-infra | 10:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Generate symlinks during tests https://review.openstack.org/553316 | 10:55 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 10:55 |
| *** rossella_s has quit IRC | 10:56 | |
| openstackgerrit | eldad marciano proposed openstack-infra/grafyaml master: Add datasource to template schema. https://review.openstack.org/548365 | 10:58 |
| *** rossella_s has joined #openstack-infra | 11:00 | |
| frickler | tobiash: technically yes, but I wouldn't feel confident enough to do so on my own. hoping either fungi, clarkb or corvus wake up early today ;) | 11:00 |
| tobiash | ok | 11:01 |
| AJaeger | frickler: thanks to day light saving, it'S one hour earlier than usual ;) | 11:04 |
| *** yamamoto has quit IRC | 11:04 | |
| tobiash | frickler, AJaeger: looking at the job failure the reason is this: http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/devstack-project-conf/tasks/main.yaml#n11 | 11:04 |
| tobiash | and that task doesn't look like it's dealing with secrets | 11:04 |
| tobiash | it uses the variable zuul.projects (the zuul var never contains secrets) | 11:04 |
| tobiash | and outputs stats of the dir /etc/<project name> | 11:05 |
| tobiash | so imho it should be safe to remove no_log there | 11:05 |
| tobiash | which would at least unbreak that | 11:06 |
| tobiash | no idea if there are further occurrences of that pattern | 11:06 |
| *** armaan_ has joined #openstack-infra | 11:09 | |
| *** armaan has quit IRC | 11:10 | |
| tobiash | AJaeger, frickler, pabelanger: https://review.openstack.org/553329 | 11:11 |
| tobiash | the no_log was just added to reduce noise so it's definitely safe to remove it | 11:11 |
| tobiash | andreaf: ^ | 11:12 |
| *** armaan has joined #openstack-infra | 11:13 | |
| *** adarazs is now known as adarazs_lunch | 11:14 | |
| *** udesale has quit IRC | 11:15 | |
| *** liuyulong has quit IRC | 11:16 | |
| *** armaan_ has quit IRC | 11:17 | |
| *** gongysh has quit IRC | 11:17 | |
| *** dtruong_ has joined #openstack-infra | 11:18 | |
| *** lpetrut_ has joined #openstack-infra | 11:18 | |
| *** lpetrut has quit IRC | 11:18 | |
| *** snapiri- has joined #openstack-infra | 11:18 | |
| *** andreas_s has quit IRC | 11:18 | |
| *** ianychoi_ has joined #openstack-infra | 11:19 | |
| *** andreas_s has joined #openstack-infra | 11:19 | |
| *** pbourke_ has joined #openstack-infra | 11:20 | |
| *** Kevin_Zheng_ has joined #openstack-infra | 11:20 | |
| *** jungleboyj_ has joined #openstack-infra | 11:20 | |
| *** zaro__ has joined #openstack-infra | 11:20 | |
| *** signed8bit has joined #openstack-infra | 11:20 | |
| *** vabada2 has joined #openstack-infra | 11:20 | |
| *** mnaser_ has joined #openstack-infra | 11:20 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 11:21 |
| *** jokke__ has joined #openstack-infra | 11:22 | |
| *** lpetrut has joined #openstack-infra | 11:22 | |
| *** vipuls has joined #openstack-infra | 11:22 | |
| *** zara_the1lemur__ has joined #openstack-infra | 11:22 | |
| *** dosaboy_ has joined #openstack-infra | 11:22 | |
| *** igormarnat_ has joined #openstack-infra | 11:23 | |
| *** rodrigod` has joined #openstack-infra | 11:23 | |
| *** gema_ has joined #openstack-infra | 11:23 | |
| *** frickler_ has joined #openstack-infra | 11:23 | |
| *** Anticime1 has joined #openstack-infra | 11:23 | |
| *** jog0 has joined #openstack-infra | 11:23 | |
| *** andreas_s has quit IRC | 11:24 | |
| *** timrchavez_ has joined #openstack-infra | 11:25 | |
| *** arxcruz has joined #openstack-infra | 11:26 | |
| *** slaweq_ has joined #openstack-infra | 11:27 | |
| *** pbourke has quit IRC | 11:27 | |
| *** pcaruana|wfh| has quit IRC | 11:27 | |
| *** diablo_rojo has quit IRC | 11:27 | |
| *** kaisers1 has quit IRC | 11:27 | |
| *** bandini has quit IRC | 11:27 | |
| *** dtruong has quit IRC | 11:27 | |
| *** igormarnat has quit IRC | 11:27 | |
| *** michaelxin has quit IRC | 11:27 | |
| *** arxcruz|off has quit IRC | 11:27 | |
| *** snapiri has quit IRC | 11:27 | |
| *** vipul has quit IRC | 11:27 | |
| *** bkero- has quit IRC | 11:27 | |
| *** jogo has quit IRC | 11:27 | |
| *** rodrigods has quit IRC | 11:27 | |
| *** dosaboy has quit IRC | 11:27 | |
| *** Kevin_Zheng has quit IRC | 11:27 | |
| *** zaro_ has quit IRC | 11:27 | |
| *** jungleboyj has quit IRC | 11:27 | |
| *** mnaser has quit IRC | 11:27 | |
| *** zara_the_lemur__ has quit IRC | 11:27 | |
| *** timrc has quit IRC | 11:27 | |
| *** frickler has quit IRC | 11:27 | |
| *** jokke_ has quit IRC | 11:27 | |
| *** Anticimex has quit IRC | 11:27 | |
| *** kzaitsev_pi has quit IRC | 11:27 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 11:27 |
| *** igormarnat_ is now known as igormarnat | 11:27 | |
| *** jungleboyj_ is now known as jungleboyj | 11:27 | |
| *** mnaser_ is now known as mnaser | 11:27 | |
| *** sshnaidm_ has joined #openstack-infra | 11:27 | |
| *** jlk` has joined #openstack-infra | 11:27 | |
| *** gfidente^2nd has joined #openstack-infra | 11:28 | |
| *** SergeyLukjanov2 has joined #openstack-infra | 11:28 | |
| *** jokke__ has quit IRC | 11:28 | |
| *** jokke__ has joined #openstack-infra | 11:28 | |
| *** pcaruana|wfh| has joined #openstack-infra | 11:28 | |
| *** jokke__ is now known as jokke_ | 11:28 | |
| *** _ari_ has quit IRC | 11:28 | |
| *** Marx314 has quit IRC | 11:28 | |
| *** signed8bit_Zzz has quit IRC | 11:28 | |
| *** vtapia has quit IRC | 11:28 | |
| *** jgr has quit IRC | 11:28 | |
| *** ianychoi has quit IRC | 11:28 | |
| *** gfidente has quit IRC | 11:28 | |
| *** liusheng has quit IRC | 11:28 | |
| *** SergeyLukjanov has quit IRC | 11:28 | |
| *** krtaylor has quit IRC | 11:28 | |
| *** gema has quit IRC | 11:28 | |
| *** kzaitsev1pi has joined #openstack-infra | 11:28 | |
| *** diablo_rojo_ has joined #openstack-infra | 11:28 | |
| *** lpetrut_ has quit IRC | 11:28 | |
| *** sshnaidm has quit IRC | 11:28 | |
| *** slaweq has quit IRC | 11:28 | |
| *** vabada has quit IRC | 11:28 | |
| *** jlk has quit IRC | 11:28 | |
| *** jgr has joined #openstack-infra | 11:28 | |
| *** _ari_ has joined #openstack-infra | 11:28 | |
| *** liusheng has joined #openstack-infra | 11:28 | |
| *** krtaylor has joined #openstack-infra | 11:28 | |
| *** Kevin_Zheng_ has quit IRC | 11:28 | |
| *** Kevin_Zheng_ has joined #openstack-infra | 11:28 | |
| *** mnaser has quit IRC | 11:28 | |
| *** mnaser has joined #openstack-infra | 11:28 | |
| *** jungleboyj has quit IRC | 11:28 | |
| *** jungleboyj has joined #openstack-infra | 11:28 | |
| *** zaro__ has quit IRC | 11:28 | |
| *** zaro__ has joined #openstack-infra | 11:28 | |
| *** SergeyLukjanov2 is now known as SergeyLukjanov | 11:28 | |
| *** kaisers1 has joined #openstack-infra | 11:28 | |
| *** yee379 has quit IRC | 11:29 | |
| *** bandini has joined #openstack-infra | 11:30 | |
| *** bkero- has joined #openstack-infra | 11:30 | |
| *** michaelxin has joined #openstack-infra | 11:31 | |
| *** frickler_ is now known as frickler | 11:35 | |
| *** kjackal has quit IRC | 11:38 | |
| *** annp has quit IRC | 11:48 | |
| *** vtapia has joined #openstack-infra | 11:48 | |
| *** armaan_ has joined #openstack-infra | 11:49 | |
| *** dmellado has quit IRC | 11:50 | |
| *** pgadiya has quit IRC | 11:51 | |
| *** armaan has quit IRC | 11:52 | |
| *** stevebaker has quit IRC | 11:52 | |
| *** ldnunes has joined #openstack-infra | 11:53 | |
| *** gothicmindfood has joined #openstack-infra | 11:53 | |
| *** dmellado has joined #openstack-infra | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 12:00 | |
| *** zoli is now known as zoli|lunch | 12:02 | |
| *** zoli|lunch is now known as zoli | 12:02 | |
| *** efoley has quit IRC | 12:02 | |
| *** kjackal has joined #openstack-infra | 12:03 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Set openstack-tox-cover as voting for monasca-ui https://review.openstack.org/553368 | 12:04 |
| *** yamamoto has joined #openstack-infra | 12:04 | |
| *** trown|outtypewww is now known as trown|ruck | 12:04 | |
| *** dtantsur|afk is now known as dtantsur | 12:04 | |
| *** dmellado has quit IRC | 12:05 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/550130 | 12:05 |
| *** odyssey4me has quit IRC | 12:05 | |
| *** odyssey4me has joined #openstack-infra | 12:05 | |
| *** yamamoto has quit IRC | 12:09 | |
| *** pcaruana|wfh| is now known as pcaruana | 12:14 | |
| *** edmondsw has joined #openstack-infra | 12:15 | |
| *** dprince has joined #openstack-infra | 12:15 | |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** claudiub|2 has joined #openstack-infra | 12:20 | |
| *** jpena is now known as jpena|lunch | 12:21 | |
| *** claudiub has quit IRC | 12:23 | |
| *** dmellado has joined #openstack-infra | 12:23 | |
| *** witek has quit IRC | 12:24 | |
| *** witek has joined #openstack-infra | 12:24 | |
| tobberydberg | Hi folks! Trying to figure out permissions for channel #openstack-publiccloud ... seanhandley has the permissions that I would like to have for that channel. Is this configurable somewhere that I can't find and someone can point me in the right direction, or is this something that just a few people can configure? | 12:27 |
| *** snapiri- is now known as snapiri | 12:27 | |
| *** snapiri has quit IRC | 12:27 | |
| *** snapiri has joined #openstack-infra | 12:28 | |
| *** lucasagomes is now known as lucas-hungry | 12:28 | |
| *** dmellado has quit IRC | 12:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Migrate Sahara projects to Storyboard https://review.openstack.org/552651 | 12:31 |
| mnaser | tobberydberg: anything in particular you need permissions for to do? | 12:31 |
| tobberydberg | setting topic mnaser | 12:35 |
| *** rlandy has joined #openstack-infra | 12:35 | |
| tobberydberg | seanhandley gave me +o earlier, but would be nice to get that without asking him | 12:35 |
| *** sshnaidm_ is now known as sshnaidm | 12:35 | |
| *** armaan_ has quit IRC | 12:36 | |
| *** chrisyang_0660 has quit IRC | 12:37 | |
| *** armaan has joined #openstack-infra | 12:37 | |
| *** gfidente^2nd is now known as gfidente | 12:37 | |
| e0ne | hi team. could anybody please help me with horizon jobs? | 12:41 |
| *** snapiri has quit IRC | 12:41 | |
| e0ne | e.g. https://review.openstack.org/#/c/522556/ horizon-dsvm-tempest-plugi POST_FAILURE | 12:41 |
| *** snapiri has joined #openstack-infra | 12:41 | |
| e0ne | rechecks didn't help. There are the same issues with other patches | 12:42 |
| *** efoley has joined #openstack-infra | 12:42 | |
| *** mikal has joined #openstack-infra | 12:42 | |
| *** psachin has quit IRC | 12:44 | |
| *** bhujay has joined #openstack-infra | 12:44 | |
| AJaeger | e0ne: https://review.openstack.org/553329 should fix the problem | 12:45 |
| *** mikal_ has quit IRC | 12:45 | |
| e0ne | AJaeger: thanks for the information | 12:45 |
| *** armaan has quit IRC | 12:45 | |
| *** armaan has joined #openstack-infra | 12:45 | |
| *** r-daneel has joined #openstack-infra | 12:46 | |
| *** hrubi has quit IRC | 12:47 | |
| *** kgiusti has joined #openstack-infra | 12:47 | |
| *** signed8bit is now known as signed8bit_Zzz | 12:47 | |
| *** dizquierdo has quit IRC | 12:48 | |
| *** hrubi has joined #openstack-infra | 12:49 | |
| frickler | e0ne: AJaeger: yes, there is a known issue, seems to mainly affect jobs based on the new zuulv3 devstack job. but that patch is only a partial fix at best and does still suffer from a lot of post_failures | 12:49 |
| *** dayou has quit IRC | 12:50 | |
| frickler | e0ne: the root cause seem to be recent changes to zuul, waiting for our zuul experts to show up and debug further | 12:51 |
| *** david-lyle has joined #openstack-infra | 12:52 | |
| *** ldnunes has quit IRC | 12:53 | |
| *** ldnunes has joined #openstack-infra | 12:54 | |
| *** signed8bit_Zzz is now known as signed8bit | 12:55 | |
| *** jamesmcarthur has joined #openstack-infra | 12:57 | |
| *** dave-mccowan has joined #openstack-infra | 13:01 | |
| *** dayou has joined #openstack-infra | 13:03 | |
| *** andreas_s has joined #openstack-infra | 13:03 | |
| *** alexchadin has joined #openstack-infra | 13:03 | |
| *** Goneri has joined #openstack-infra | 13:08 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import python_venv_build ansible role https://review.openstack.org/553392 | 13:08 |
| *** dsariel has quit IRC | 13:08 | |
| *** eharney has joined #openstack-infra | 13:09 | |
| *** myoung|afk is now known as myoung|rover | 13:09 | |
| *** mriedem has joined #openstack-infra | 13:09 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import python_venv_build ansible role https://review.openstack.org/553392 | 13:09 |
| *** zoli is now known as zoli|wfh | 13:10 | |
| *** zoli|wfh is now known as zoli | 13:10 | |
| *** david-lyle has quit IRC | 13:12 | |
| *** panda|off is now known as panda | 13:13 | |
| *** jamesmcarthur has quit IRC | 13:14 | |
| *** sree has joined #openstack-infra | 13:14 | |
| frickler | tobiash: dulek: is looks like the apache hardlinks might indeed be the root cause of the current issues and the other stuff just masking it http://logs.openstack.org/86/547986/7/check/tempest-full/1338109/ara/result/29d01451-fa45-46e1-a27e-11b54d96d3b7/ | 13:15 |
| *** david-lyle has joined #openstack-infra | 13:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add base jobs for python_venv_build ansible role https://review.openstack.org/553396 | 13:15 |
| dulek | frickler: Hah, a point to intuition! ;) | 13:15 |
| *** adarazs_lunch is now known as adarazs | 13:18 | |
| *** yamamoto has quit IRC | 13:18 | |
| *** sree has quit IRC | 13:18 | |
| *** eernst has joined #openstack-infra | 13:19 | |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| *** dmellado has joined #openstack-infra | 13:19 | |
| *** dmellado has quit IRC | 13:20 | |
| *** rosmaita has joined #openstack-infra | 13:21 | |
| *** jcoufal_ has joined #openstack-infra | 13:21 | |
| Shrews | looks like the devstack apache-logs-conf role is also using no_log around a with_items task. i wonder if we need to take 'when' usage into account before clearing out the item | 13:23 |
| *** ianychoi_ is now known as ianychoi | 13:23 | |
| *** yamamoto has quit IRC | 13:23 | |
| Shrews | corvus: tobiash: ^^ | 13:23 |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| Shrews | http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/apache-logs-conf/tasks/main.yaml#n27 | 13:24 |
| *** jcoufal has quit IRC | 13:24 | |
| *** hashar has quit IRC | 13:24 | |
| *** david-lyle has quit IRC | 13:25 | |
| *** gouthamr_ has joined #openstack-infra | 13:26 | |
| *** gouthamr_ is now known as gouthamr | 13:26 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** snapiri has quit IRC | 13:29 | |
| *** ralonsoh_ has joined #openstack-infra | 13:32 | |
| *** snapiri has joined #openstack-infra | 13:32 | |
| *** jamesmcarthur has joined #openstack-infra | 13:32 | |
| Shrews | i wonder if we can clear the data out of the item itself, rather than replace it with a dict | 13:33 |
| *** dave-mccowan has quit IRC | 13:34 | |
| *** ricardoas has quit IRC | 13:35 | |
| *** ralonsoh has quit IRC | 13:35 | |
| *** felipemonteiro__ has joined #openstack-infra | 13:36 | |
| *** gema_ is now known as gema | 13:38 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:38 | |
| *** gema has quit IRC | 13:38 | |
| *** gema has joined #openstack-infra | 13:38 | |
| *** myoung|rover is now known as myoung|logdiver | 13:39 | |
| *** myoung|logdiver is now known as myoung|rover | 13:39 | |
| tosky | can at least anyone send a notification to tell people to not recheck while this is investigated? | 13:41 |
| *** hashar has joined #openstack-infra | 13:41 | |
| *** felipemonteiro__ has quit IRC | 13:42 | |
| *** lucas-hungry is now known as lucasagomes | 13:42 | |
| *** links has quit IRC | 13:42 | |
| *** eernst has quit IRC | 13:43 | |
| *** udesale has joined #openstack-infra | 13:43 | |
| *** felipemonteiro_ has quit IRC | 13:45 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:45 | |
| *** dsariel has joined #openstack-infra | 13:47 | |
| *** r-daneel has quit IRC | 13:48 | |
| *** zhipeng has joined #openstack-infra | 13:51 | |
| *** zoli is now known as zoli|afk | 13:52 | |
| *** zoli|afk is now known as zoli | 13:52 | |
| *** janki has joined #openstack-infra | 13:52 | |
| mnaser | tosky: probably a good idea bevcause i was just going to ask | 13:53 |
| mnaser | infra-root: ^ ? | 13:53 |
| *** erlon has quit IRC | 13:53 | |
| *** hashar has quit IRC | 13:54 | |
| *** jamesmcarthur has quit IRC | 14:00 | |
| *** esberglu has joined #openstack-infra | 14:00 | |
| *** bhujay has quit IRC | 14:01 | |
| *** ihrachys_ is now known as ihrachys | 14:01 | |
| *** gongysh has joined #openstack-infra | 14:01 | |
| *** annp has joined #openstack-infra | 14:03 | |
| *** jaosorior has quit IRC | 14:03 | |
| *** ralonsoh__ has joined #openstack-infra | 14:04 | |
| fungi | i haven't been following closely... is it just impacting tempest-based jobs at the moment? | 14:04 |
| *** ralonsoh__ is now known as ralonsoh | 14:04 | |
| *** neiljerram has joined #openstack-infra | 14:05 | |
| fungi | something like #status alert POST_FAILURE results on Tempest-based jobs since the most recent Zuul security fixes are being investigated; rechecking those won't help for now but we'll keep you posted once a solution is identified | 14:05 |
| tosky | that should be enough, thanks | 14:06 |
| fungi | #status alert POST_FAILURE results on Tempest-based jobs since the most recent Zuul security fixes are being investigated; rechecking those won't help for now but we'll keep you posted once a solution is identified | 14:06 |
| openstackstatus | fungi: sending alert | 14:06 |
| *** ralonsoh_ has quit IRC | 14:07 | |
| Shrews | fungi: i don't know the full extent of what is affected, but i've been looking at the devstack-based job failure to identify the issue | 14:07 |
| *** kiennt26 has joined #openstack-infra | 14:07 | |
| -openstackstatus- NOTICE: POST_FAILURE results on Tempest-based jobs since the most recent Zuul security fixes are being investigated; rechecking those won't help for now but we'll keep you posted once a solution is identified | 14:09 | |
| *** ChanServ changes topic to "POST_FAILURE results on Tempest-based jobs since the most recent Zuul security fixes are being investigated; rechecking those won't help for now but we'll keep you posted once a solution is identified" | 14:09 | |
| *** bobh has joined #openstack-infra | 14:10 | |
| openstackstatus | fungi: finished sending alert | 14:12 |
| rm_work | wait, so all our channel topics got replaced by ... that? :P | 14:14 |
| rm_work | I mean, it's useful I guess... but lol | 14:14 |
| *** eernst has joined #openstack-infra | 14:15 | |
| tosky | more useful than "you can't do anything"? :) | 14:15 |
| tosky | it will be restored eventually | 14:15 |
| *** rossella_s has quit IRC | 14:15 | |
| fungi | rm_work: you can opt out of that behavior by not having the statusbot (openstackstatus) join your channel | 14:16 |
| *** eernst has quit IRC | 14:16 | |
| rm_work | it's just interesting, but i guess if it restores the old channel topic eventually it's fine ;P | 14:16 |
| fungi | it will revert your topic to its normal value when we issue a #status ok | 14:16 |
| rm_work | just odd having that huge block instead of "welcome to <x>" and associated info | 14:16 |
| *** dizquierdo has joined #openstack-infra | 14:16 | |
| rm_work | though granted we don't really have any useful info >_> | 14:16 |
| fungi | s/normal/previous/ | 14:17 |
| rm_work | yeah i just never noticed it before i guess | 14:17 |
| fungi | statusbot stores the topic your channel had before updating, and then sets it back again when done | 14:17 |
| *** rossella_s has joined #openstack-infra | 14:17 | |
| *** eernst has joined #openstack-infra | 14:18 | |
| *** yamamoto has joined #openstack-infra | 14:20 | |
| neiljerram | Can anyone advise where I should look if I see "process-returncode: fail" in a CI run, as for example here: http://logs.openstack.org/18/552518/5/gate/openstack-tox-py35/b174533/testr_results.html.gz | 14:23 |
| neiljerram | ("process-returncode" is not the name of any actual test) | 14:24 |
| AJaeger | rm_work: https://wiki.openstack.org/wiki/Infrastructure_Status contains all the message we log, many of them are alerts | 14:25 |
| rm_work | yeah i just didn't realize it changes all the channel topics automatically ;P | 14:25 |
| rm_work | just struck me as interesting/humorous | 14:25 |
| *** yamamoto has quit IRC | 14:25 | |
| sc` | neiljerram: http://logs.openstack.org/18/552518/5/gate/openstack-tox-py35/b174533/job-output.txt.gz suggests an invokation error. have you seen that yet? | 14:26 |
| *** zhipeng has quit IRC | 14:27 | |
| *** bobh has quit IRC | 14:27 | |
| sc` | s/invokation/invocation/ ENOCAFFEINE | 14:27 |
| *** ykarel is now known as ykarel|away | 14:27 | |
| neiljerram | sc`, yes, I saw that too; but still not really understanding what it means, I'm afraid. | 14:28 |
| *** dmellado has joined #openstack-infra | 14:28 | |
| *** dsariel has quit IRC | 14:30 | |
| *** r-daneel has joined #openstack-infra | 14:30 | |
| *** armaan has quit IRC | 14:31 | |
| *** gongysh has quit IRC | 14:31 | |
| *** germs has joined #openstack-infra | 14:33 | |
| *** germs has quit IRC | 14:33 | |
| *** germs has joined #openstack-infra | 14:33 | |
| *** germs has quit IRC | 14:33 | |
| *** zoli is now known as zoli|wfh | 14:35 | |
| *** ykarel|away has quit IRC | 14:36 | |
| *** germs has joined #openstack-infra | 14:36 | |
| corvus | AJaeger, fungi, frickler: i expect a zuul fix to take a few hours. we can either land a change to remove no_log from stat, or we can revert the security fix, or we can just wait for it. thoughts? | 14:37 |
| *** yamahata has joined #openstack-infra | 14:39 | |
| AJaeger | corvus: I have not have time to follow closely and can really say how much it hurts us. It looks like we have time to wait for the fix. | 14:41 |
| frickler | corvus: I'm not sure which places would need the no_log removed, so I guess reverting would be sensible. but it's also almost eod for me, so if it gets fixed within the next 12h, that would be fine for me, too | 14:41 |
| *** florianf has joined #openstack-infra | 14:43 | |
| fungi | i'm not keen on reverting security fixes, if we can help it | 14:44 |
| fungi | especially now that we've announced the details of how we were vulnerable before the fixes went into effect | 14:44 |
| sc` | neiljerram: older pip choking on the git url perhaps? feels like something more in clarkb's wheelhouse per github issues | 14:45 |
| fungi | running with the current regression seems preferable to shutting down zuul until we have a fix for the fix | 14:45 |
| *** dmellado_ has joined #openstack-infra | 14:46 | |
| *** olaph1 has quit IRC | 14:47 | |
| *** jaosorior has joined #openstack-infra | 14:47 | |
| fungi | sc`: neiljerram: is the "Exception in SubnetWatcher.start()" (followed by series of tracebacks from test_kill_agent) normal? | 14:47 |
| *** olaph has joined #openstack-infra | 14:48 | |
| neiljerram | fungi, Yes, it is expected in those tests. | 14:48 |
| sc` | neiljerram: the job-output log suggests that things are choking in the 'pip freeze' stage http://logs.openstack.org/18/552518/5/gate/openstack-tox-py35/b174533/tox/py35-5.log | 14:48 |
| neiljerram | sc`, thanks, I'll look at that. | 14:48 |
| sc` | it referenced that log file as for what step was executing. my guess is that it's _test-requirements.txt not playing along | 14:49 |
| fungi | sc`: if you're talking about the "Error when trying to get requirement for VCS system..." that's a benign warning from pip. clarkb has a pr up to make that nicer in future pip versions (if they ever review it) | 14:50 |
| sc` | ah | 14:50 |
| AJaeger | neiljerram: I'm concerend about http://logs.openstack.org/18/552518/5/gate/openstack-tox-py35/b174533/job-output.txt.gz#_2018-03-15_11_33_15_931837 | 14:50 |
| fungi | sc`: it | 14:50 |
| AJaeger | the uninstalling. | 14:50 |
| AJaeger | I would remove "-U" from "pip install" - let me send a patch | 14:50 |
| sc` | fungi: good point. however, job-output suggests it's not the warning at fault | 14:50 |
| fungi | sc`: er, it's just warning you that it can't include git url details in the pip freeze output | 14:50 |
| *** jbadiapa has quit IRC | 14:50 | |
| fungi | sc`: neiljerram: my gut says the invocation error on "setup.py test --slowest --testr-args=" may be related to the stestr transition. mtreinish might have some ideas | 14:52 |
| sc` | that stood out as odd to me, and came to a similar point earlier | 14:52 |
| neiljerram | AJaeger, thanks for spotting that; I don't recall anything in that test's setup that is intentionally uninstalling anything... | 14:52 |
| AJaeger | neiljerram: it's a side effect of how we setup the job | 14:53 |
| AJaeger | neiljerram: and you not using constraints. Do you want to use constraints? | 14:53 |
| sc` | in retrospect, the invocation would be a good place to drill down on | 14:53 |
| mtreinish | fungi, sc`: I mean that's using the pbr testr entrypoint to run tests | 14:54 |
| mtreinish | stestr isn't really used there | 14:54 |
| *** jtomasek_ has joined #openstack-infra | 14:54 | |
| mtreinish | iirc that is being deprecated/removed from pbr at some point | 14:54 |
| fungi | mtreinish: i was wondering whether it should be, but honestly don't know | 14:54 |
| neiljerram | fungi, yes, I wonder if I've missed a recommended new way of doing things; I checked in another networking-* project and found use of 'ostestr_compat_shim.sh' there - which networking-calico doesn't have | 14:54 |
| mtreinish | fungi: I mean everything should be moving to stestr | 14:54 |
| *** dmellado has quit IRC | 14:54 | |
| *** dmellado_ is now known as dmellado | 14:54 | |
| *** cshastri has joined #openstack-infra | 14:54 | |
| mtreinish | fungi: that's what we updated the PTI to say | 14:55 |
| mtreinish | https://governance.openstack.org/tc/reference/pti/python.html#python-test-running | 14:55 |
| *** myoung|rover is now known as myoung|rover|brb | 14:56 | |
| neiljerram | mtreinish, thanks, I'll check that doc and look at transitioning networking-calico accordingly | 14:56 |
| *** eharney has quit IRC | 14:56 | |
| fungi | cool, i wasn't sure what that ended up looking like in job logs and honestly haven't been paying attention to the details of the transition patches | 14:56 |
| AJaeger | neiljerram: https://review.openstack.org/553425 | 14:56 |
| *** eharney has joined #openstack-infra | 14:56 | |
| *** bobh has joined #openstack-infra | 14:57 | |
| sc` | where is the partyparrot emoji when you need it? | 14:58 |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Custom Template Type https://review.openstack.org/544305 | 14:58 |
| * fungi is the original partyparrot, but didn't know he had an emoji | 14:58 | |
| sc` | RIP your eyes: http://cultofthepartyparrot.com/ | 14:59 |
| fungi | it cannot be unseen | 14:59 |
| sc` | quite. | 15:00 |
| *** felipemonteiro_ has quit IRC | 15:01 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:01 | |
| mgagne | which channel is the best place for support for the marketplace on openstack.org ? | 15:01 |
| *** beagles is now known as beagles_food | 15:02 | |
| neiljerram | AJaeger, thanks, will see how that does. FYI the _test-requirements.txt naming dates back to https://git.openstack.org/cgit/openstack/networking-calico/commit/?id=05e2deb6cf74c6620df14108f65ef29acf658b3c, which was motivated at the time by a devstack/pip issue (https://bugs.launchpad.net/devstack/+bug/1540328) | 15:02 |
| openstack | Launchpad bug 1540328 in Magnum UI "Devstack cannot install Horizon plugins using pip version > 8" [High,Fix released] - Assigned to Shu Muto (shu-mutou) | 15:02 |
| clarkb | have we confirmed a no_log fixes the apache thing? just to be sure it is a bug with the no_log chnage? | 15:03 |
| clarkb | *have we confirmed removing no_log | 15:03 |
| *** bobh has quit IRC | 15:03 | |
| *** annp has quit IRC | 15:03 | |
| AJaeger | neiljerram: let's test it - this all looks strange... | 15:04 |
| AJaeger | neiljerram: with my change it should be just fine (just read the bug) | 15:04 |
| neiljerram | AJaeger, yep; I'll keep an eye and this and come back here if needed. | 15:05 |
| *** jbadiapa has joined #openstack-infra | 15:06 | |
| *** dsariel has joined #openstack-infra | 15:06 | |
| *** bobh has joined #openstack-infra | 15:07 | |
| *** sree has joined #openstack-infra | 15:07 | |
| *** sree_ has joined #openstack-infra | 15:08 | |
| *** beagles_food is now known as beagles | 15:09 | |
| *** sree_ is now known as Guest6416 | 15:09 | |
| *** bobh has quit IRC | 15:09 | |
| *** bobh has joined #openstack-infra | 15:09 | |
| *** udesale has quit IRC | 15:10 | |
| *** olaph1 has joined #openstack-infra | 15:11 | |
| *** sree has quit IRC | 15:12 | |
| *** olaph has quit IRC | 15:12 | |
| *** tmorin1 has joined #openstack-infra | 15:13 | |
| *** r-daneel_ has joined #openstack-infra | 15:14 | |
| *** tmorin has quit IRC | 15:14 | |
| *** r-daneel has quit IRC | 15:14 | |
| *** r-daneel_ is now known as r-daneel | 15:14 | |
| *** stevebaker has joined #openstack-infra | 15:14 | |
| *** eernst has quit IRC | 15:15 | |
| *** krtaylor has quit IRC | 15:16 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:17 | |
| *** armaan has joined #openstack-infra | 15:21 | |
| *** felipemonteiro_ has quit IRC | 15:21 | |
| *** yamamoto has joined #openstack-infra | 15:21 | |
| *** yamamoto has quit IRC | 15:27 | |
| mgagne | we would like to perform a 1h maintenance on inap-mtl01 region, would tonight be too early? API will be unavailable, existing instances will continue to work fine. | 15:28 |
| mgagne | tonight -> montreal timezone | 15:29 |
| clarkb | mgagne: that should be fine | 15:31 |
| *** iyamahat has joined #openstack-infra | 15:31 | |
| mgagne | is 6:00pm fine with you? | 15:32 |
| pabelanger | tobiash: AJaeger: -1 on 553329 left a comment | 15:35 |
| clarkb | mgagne: ya that should be fine. | 15:35 |
| *** dave-mccowan has joined #openstack-infra | 15:36 | |
| openstackgerrit | Merged openstack-infra/zuul master: Revert "Don't store references to secret objects from jobs" https://review.openstack.org/553147 | 15:37 |
| *** myoung|rover|brb is now known as myoung|rover | 15:38 | |
| *** jtomasek_ has quit IRC | 15:39 | |
| *** lpetrut_ has joined #openstack-infra | 15:43 | |
| *** lpetrut has quit IRC | 15:43 | |
| *** yamahata has quit IRC | 15:43 | |
| *** kiennt26 has quit IRC | 15:44 | |
| *** iyamahat has quit IRC | 15:49 | |
| *** lpetrut_ has quit IRC | 15:49 | |
| *** agopi has joined #openstack-infra | 15:51 | |
| *** gongysh has joined #openstack-infra | 15:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: improve no_log test https://review.openstack.org/553450 | 15:53 |
| *** kmalloc has joined #openstack-infra | 15:53 | |
| *** eernst has joined #openstack-infra | 15:53 | |
| *** mahatic has joined #openstack-infra | 15:53 | |
| *** agopi has quit IRC | 15:55 | |
| *** mahatic_ has joined #openstack-infra | 15:56 | |
| *** hamzy has quit IRC | 15:56 | |
| *** tosky has quit IRC | 15:57 | |
| *** dtruong_ has quit IRC | 15:58 | |
| *** dtruong has joined #openstack-infra | 15:58 | |
| *** mahatic has quit IRC | 15:59 | |
| *** ralonsoh_ has joined #openstack-infra | 16:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: improve no_log test https://review.openstack.org/553450 | 16:01 |
| *** gfidente has quit IRC | 16:01 | |
| *** gfidente has joined #openstack-infra | 16:02 | |
| *** gfidente has joined #openstack-infra | 16:02 | |
| *** armaan has quit IRC | 16:02 | |
| *** florianf has quit IRC | 16:02 | |
| *** alexchadin has quit IRC | 16:03 | |
| *** armaan has joined #openstack-infra | 16:03 | |
| *** armaan has quit IRC | 16:03 | |
| *** harlowja has joined #openstack-infra | 16:03 | |
| *** armaan has joined #openstack-infra | 16:03 | |
| *** _alastor_ has joined #openstack-infra | 16:04 | |
| *** ralonsoh has quit IRC | 16:04 | |
| *** iyamahat has joined #openstack-infra | 16:05 | |
| *** armaan has quit IRC | 16:07 | |
| *** armaan has joined #openstack-infra | 16:08 | |
| *** gongysh has quit IRC | 16:08 | |
| fungi | mgagne: i don't know that the marketplace devs have any one particular venue in irc, but jamesmcarthur (who isn't in channel at the moment but tends to hang out in here regularly) can probably answer questions | 16:09 |
| *** electrofelix has quit IRC | 16:11 | |
| *** alexchadin has joined #openstack-infra | 16:11 | |
| *** gfidente has quit IRC | 16:13 | |
| *** jcoufal has joined #openstack-infra | 16:14 | |
| *** gfidente has joined #openstack-infra | 16:14 | |
| *** gfidente has quit IRC | 16:14 | |
| *** gfidente has joined #openstack-infra | 16:14 | |
| *** jcoufal_ has quit IRC | 16:15 | |
| *** myoung|rover is now known as myoung|lunch | 16:15 | |
| *** kjackal has quit IRC | 16:18 | |
| *** Guest6416 has quit IRC | 16:20 | |
| *** sree has joined #openstack-infra | 16:20 | |
| *** jaosorior has quit IRC | 16:21 | |
| *** kjackal has joined #openstack-infra | 16:21 | |
| *** yamahata has joined #openstack-infra | 16:22 | |
| *** yamamoto has joined #openstack-infra | 16:23 | |
| clarkb | you can probably file an isse on the github for the www site? | 16:23 |
| clarkb | er I guess they use lp | 16:24 |
| clarkb | but ya that | 16:24 |
| fungi | mgagne: yeah, https://bugs.launchpad.net/openstack-org/+filebug if you want to report a bug with it | 16:24 |
| mtreinish | fungi, clarkb: I've got a question, do you know why it says personal user here: https://www.openstack.org/summit/vancouver-2018/summit-schedule/speakers/108 | 16:24 |
| *** sree has quit IRC | 16:25 | |
| mtreinish | while that's true for my openstack usage, it's not really my employer... | 16:25 |
| fungi | mtreinish: maybe they meant "personable user"? ;) | 16:25 |
| *** hongbin has joined #openstack-infra | 16:25 | |
| clarkb | mtreinish: maybe thats part of your AUC status? eg run cloud for your personal use? | 16:25 |
| fungi | i'll ask whether that's taken from your foundation member profile, foundation speaker profile, user survey, or what | 16:25 |
| mtreinish | I probably put that on the user survey | 16:26 |
| mtreinish | for company, because the survey kinda assumes you use it at a company | 16:26 |
| *** ykarel|away has joined #openstack-infra | 16:26 | |
| fungi | mtreinish: oh! https://www.openstack.org/community/members/profile/5714/matthew-treinish | 16:27 |
| fungi | you have two current affiliations, one of which is "personal user" | 16:27 |
| *** e0ne has quit IRC | 16:27 | |
| *** masber has joined #openstack-infra | 16:28 | |
| mtreinish | oh, that would do it. I definitely didn't do that explicitly though. I wonder if it did that automagically from the user survey | 16:28 |
| *** yamamoto has quit IRC | 16:29 | |
| fungi | i wouldn't be surprised | 16:29 |
| clarkb | both are current though so the bug is probably not concatenating all current affiliations together on that page | 16:30 |
| fungi | yeah, it probably just takes the first one returned from its db query or something | 16:30 |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Thanks. | 16:35 |
| *** ramishra has quit IRC | 16:38 | |
| *** andreas_s has quit IRC | 16:38 | |
| *** gongysh has joined #openstack-infra | 16:38 | |
| *** tosky has joined #openstack-infra | 16:39 | |
| *** gongysh has quit IRC | 16:39 | |
| *** mgoddard_ has joined #openstack-infra | 16:40 | |
| *** agopi has joined #openstack-infra | 16:40 | |
| *** andreas_s has joined #openstack-infra | 16:43 | |
| *** lpetrut has joined #openstack-infra | 16:43 | |
| *** myoung|lunch is now known as myoung|rover | 16:45 | |
| *** janki has quit IRC | 16:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix no_log bug with result lists https://review.openstack.org/553450 | 16:46 |
| *** efoley has quit IRC | 16:46 | |
| *** alexchadin has quit IRC | 16:48 | |
| *** gfidente has quit IRC | 16:51 | |
| *** ykarel|away has quit IRC | 16:52 | |
| *** andreas_s has quit IRC | 16:52 | |
| *** efoley has joined #openstack-infra | 16:52 | |
| *** wolverineav has joined #openstack-infra | 16:53 | |
| *** masber has quit IRC | 16:53 | |
| *** wolverineav has quit IRC | 16:53 | |
| *** gfidente has joined #openstack-infra | 16:53 | |
| *** gfidente has joined #openstack-infra | 16:53 | |
| *** wolverineav has joined #openstack-infra | 16:53 | |
| *** trown|ruck is now known as trown|lunch | 16:55 | |
| *** lpetrut has quit IRC | 16:57 | |
| *** dhajare has quit IRC | 16:57 | |
| *** dhajare has joined #openstack-infra | 16:58 | |
| *** dhajare has quit IRC | 16:58 | |
| *** olaph1 has quit IRC | 16:59 | |
| *** tmorin1 has quit IRC | 16:59 | |
| *** olaph has joined #openstack-infra | 17:00 | |
| *** lpetrut has joined #openstack-infra | 17:03 | |
| *** iyamahat_ has joined #openstack-infra | 17:04 | |
| *** eernst has quit IRC | 17:04 | |
| *** iyamahat has quit IRC | 17:04 | |
| *** panda is now known as panda|off | 17:05 | |
| *** ralonsoh__ has joined #openstack-infra | 17:06 | |
| *** eernst has joined #openstack-infra | 17:06 | |
| *** eernst has quit IRC | 17:07 | |
| *** eernst has joined #openstack-infra | 17:07 | |
| *** ykarel|away has joined #openstack-infra | 17:09 | |
| *** tosky has quit IRC | 17:09 | |
| *** ralonsoh_ has quit IRC | 17:09 | |
| *** harlowja has quit IRC | 17:09 | |
| *** jcoufal has quit IRC | 17:12 | |
| *** hamzy has joined #openstack-infra | 17:13 | |
| *** ykarel|away has quit IRC | 17:14 | |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix no_log bug with result lists https://review.openstack.org/553450 | 17:15 |
| *** mahatic_ has quit IRC | 17:16 | |
| *** ralonsoh__ has quit IRC | 17:16 | |
| *** hamzy_ has joined #openstack-infra | 17:16 | |
| clarkb | corvus: ^ executors need to be restarted but once that is done apache copying should be happy? | 17:17 |
| *** armaan has quit IRC | 17:17 | |
| *** armaan has joined #openstack-infra | 17:18 | |
| *** gouthamr has quit IRC | 17:18 | |
| Shrews | clarkb: do they? it's a change to the ansible callback module, so i'd expect that to be picked up automatically | 17:18 |
| *** gouthamr has joined #openstack-infra | 17:18 | |
| *** armaan has quit IRC | 17:19 | |
| Shrews | unless the executors copy them somewhere first on startup | 17:19 |
| pabelanger | yah, I think that is what happens | 17:19 |
| Shrews | ah | 17:19 |
| *** hamzy has quit IRC | 17:19 | |
| *** armaan has joined #openstack-infra | 17:19 | |
| Shrews | /ignore Shrews | 17:19 |
| pabelanger | I could be wrong :) | 17:20 |
| *** dizquierdo has quit IRC | 17:21 | |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n1730 | 17:22 |
| pabelanger | I think that is the code to copy them on startup | 17:22 |
| clarkb | ya I think we had to restart them to pick up the initial fix | 17:23 |
| pabelanger | I can help out here in a few moments | 17:23 |
| *** e0ne has joined #openstack-infra | 17:23 | |
| clarkb | the other change (revert of a config update) affects the scheduler which we never restarted to pick up in the first place so I think it is just the executors that we want to restart | 17:25 |
| *** yamamoto has joined #openstack-infra | 17:25 | |
| *** felipemonteiro__ has quit IRC | 17:25 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:26 | |
| corvus | yep | 17:26 |
| *** hamzy__ has joined #openstack-infra | 17:26 | |
| clarkb | I'm going to check if the new code is installed on the executors now | 17:26 |
| clarkb | doesn't look like it | 17:27 |
| pabelanger | okay, coffee obtained. What can I do? | 17:27 |
| clarkb | next ansible will start in 3 minutes | 17:27 |
| clarkb | so probably easiest to just wait for that to start on its own rather than do it out of band | 17:28 |
| clarkb | then in ~10 minutes we restart all of the executors | 17:28 |
| *** agopi has quit IRC | 17:28 | |
| corvus | i'm going to afk for a bit; i'll leave it to y'all | 17:28 |
| corvus | i've just been doing this: | 17:28 |
| corvus | 2018 2018-03-14T21:23:59+0000 sudo ansible -f 10 'ze*' -m shell -a 'systemctl stop zuul-executor' | 17:28 |
| corvus | 2019 2018-03-14T21:24:23+0000 sudo ansible -f 10 'ze*' -m shell -a 'ps aux|grep zuul-exec' | 17:28 |
| corvus | 2020 2018-03-14T21:31:55+0000 sudo ansible -f 10 'ze*' -m shell -a 'systemctl start zuul-executor' | 17:28 |
| corvus | where, obviously, the middle command is just me waiting for everything to die | 17:28 |
| *** SumitNaiksatam has joined #openstack-infra | 17:29 | |
| *** hamzy_ has quit IRC | 17:29 | |
| pabelanger | wfm | 17:30 |
| *** yamamoto has quit IRC | 17:30 | |
| *** iyamahat_ has quit IRC | 17:31 | |
| Shrews | we could just do manual copies of the callback module from /opt/zuul/zuul/ansible/callback to /var/lib/zuul/ansible/zuul/ansible/callback and chown zuul:zuul | 17:31 |
| Shrews | would be quicker | 17:31 |
| *** iyamahat has joined #openstack-infra | 17:31 | |
| *** jlk` is now known as jlk | 17:31 | |
| clarkb | I don't mind restarting and having zuul do it. Its probably an extra 10 minutes or so and this has been happening for over 12 hours now I think | 17:32 |
| clarkb | (and this way we are sure zuul is happy with it) | 17:33 |
| Shrews | *nod* | 17:33 |
| *** krtaylor has joined #openstack-infra | 17:36 | |
| evrardjp | It would be nice if I could have some eyes on https://review.openstack.org/#/c/546678/ https://review.openstack.org/#/c/552861/ and https://review.openstack.org/#/c/553392/ | 17:43 |
| evrardjp | adding new things! | 17:43 |
| *** imacdonn has quit IRC | 17:44 | |
| clarkb | pabelanger: ok I think git and pip updated on the executors but I am double checking it all before we start the restarts | 17:44 |
| *** imacdonn has joined #openstack-infra | 17:44 | |
| fungi | clarkb: we're approaching something like 22 hours at this point actually | 17:45 |
| *** jamesmcarthur has joined #openstack-infra | 17:45 | |
| clarkb | pbr freeze doesn't work with python3? | 17:45 |
| fungi | was first noticed in jobs around 20:00z | 17:45 |
| fungi | clarkb: how are you running pbr freeze? | 17:46 |
| clarkb | fungi: just `pbr freeze | grep zuul` which is going to look at python2. | 17:46 |
| pabelanger | clarkb: yah, looks like it | 17:46 |
| fungi | you likely need to `python3 -m pbr freeze` or something | 17:46 |
| fungi | hrm, nope that doesn't seem to do it | 17:46 |
| clarkb | /usr/bin/python: No module named pbr.__main__; 'pbr' is a package and cannot be directly executed | 17:46 |
| fungi | yeah, that's what i got too | 17:47 |
| clarkb | I'm going to try run it under python3 interpreter as a command not module | 17:47 |
| pabelanger | however, does look like a little issue with puppet-zuul: http://paste.openstack.org/show/702017/ | 17:47 |
| *** shardy has quit IRC | 17:47 | |
| clarkb | that works | 17:47 |
| pabelanger | can push up a patch in a few moments | 17:47 |
| fungi | clarkb: yep, works | 17:47 |
| fungi | `python3 /usr/local/bin/pbr freeze` gets me what we want | 17:47 |
| clarkb | I confirm that zuul is updated to the correct version on all 10 executor nodes | 17:49 |
| *** jpich has quit IRC | 17:49 | |
| clarkb | I'm going to run corvus' command above to stop all of the executors now | 17:49 |
| clarkb | ok they have all been told to stop. Now we wait for them to stop | 17:50 |
| *** r-daneel_ has joined #openstack-infra | 17:50 | |
| *** cshastri has quit IRC | 17:51 | |
| *** r-daneel has quit IRC | 17:51 | |
| *** r-daneel_ is now known as r-daneel | 17:51 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Fix Not removing directory; use 'force' to override https://review.openstack.org/553509 | 17:52 |
| pabelanger | clarkb: fungi: should clean up out zuul warnings^ | 17:53 |
| pabelanger | for puppet | 17:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Fix Not removing directory; use 'force' to override https://review.openstack.org/553509 | 17:55 |
| *** slaweq_ has quit IRC | 17:55 | |
| *** slaweq has joined #openstack-infra | 17:55 | |
| *** lucasagomes is now known as lucas-afk | 17:56 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:57 | |
| clarkb | starting zuul executors again | 17:58 |
| *** mgoddard_ has quit IRC | 17:58 | |
| clarkb | alright we should be running with the fix in place now | 17:58 |
| clarkb | I guess we watch some jobs and once confirmed working status ok? | 17:58 |
| fungi | sounds right | 17:59 |
| *** felipemonteiro__ has quit IRC | 17:59 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: fixes wrong cast on location service for updatemap/image https://review.openstack.org/553539 | 17:59 |
| fungi | watching openstack/nova 499612,7 at the top of the integrated gate since it's just started some tempest jobs | 18:00 |
| fungi | though given the failures were in the post phase, it'll take a while | 18:01 |
| clarkb | http://zuul.openstack.org/stream.html?uuid=8803465cd76446d6bf98c5603c08d586&logfile=console.log in particular should give us good feedback | 18:01 |
| fungi | did we know of any faster jobs exhibiting the same issues? | 18:01 |
| pabelanger | ack | 18:01 |
| clarkb | fungi: no I think it was part of the devstack/tempest job config | 18:02 |
| clarkb | fungi: and not sure if any other jobs did the right set of steps to trip it | 18:02 |
| fungi | so we're not going to know right away | 18:02 |
| *** trown|lunch is now known as trown|ruck | 18:02 | |
| *** zoli|wfh is now known as zoli|gone | 18:03 | |
| *** zoli|gone is now known as zoli | 18:03 | |
| *** derekh has quit IRC | 18:04 | |
| clarkb | the necessary combo was no_log with a list of items | 18:04 |
| *** david-lyle has joined #openstack-infra | 18:05 | |
| *** iyamahat_ has joined #openstack-infra | 18:10 | |
| dmsimard | When the dust has settled, I'd like to give a shot at merging and babysitting the logs-dev.o.o ara implementation | 18:10 |
| *** iyamahat has quit IRC | 18:11 | |
| pabelanger | dmsimard: sure, I can help support if needed | 18:13 |
| dmsimard | I'll reach out if it breaks something and I can't figure it out for sure, I just don't want to (potentially) break things in the middle of other broken things | 18:13 |
| clarkb | it should be fairly safe since logs.o.o is a separate vhost | 18:14 |
| *** jpena is now known as jpena|off | 18:14 | |
| openstackgerrit | Trevor McCasland proposed openstack-infra/subunit2sql master: Add attachments flag to get_test_runs_by_status https://review.openstack.org/552152 | 18:15 |
| dmsimard | clarkb: yeah, the only thing I'm worried about is one of the python deps breaking os-loganalyze which is used for logs.o.o as well | 18:16 |
| *** Adri2000 has quit IRC | 18:16 | |
| dmsimard | I mean, I'm fairly confident it's not going to break things -- I'm just saying if something breaks, it's probably that. | 18:16 |
| *** Adri2000 has joined #openstack-infra | 18:18 | |
| *** harlowja has joined #openstack-infra | 18:20 | |
| *** alexchadin has joined #openstack-infra | 18:21 | |
| *** yamamoto has joined #openstack-infra | 18:27 | |
| *** david-lyle has quit IRC | 18:29 | |
| *** wolverineav has quit IRC | 18:31 | |
| *** wolverineav has joined #openstack-infra | 18:31 | |
| *** yamamoto has quit IRC | 18:33 | |
| *** david-lyle has joined #openstack-infra | 18:33 | |
| *** rossella_s has quit IRC | 18:33 | |
| *** rossella_s has joined #openstack-infra | 18:34 | |
| *** wolverineav has quit IRC | 18:35 | |
| *** wolverineav has joined #openstack-infra | 18:37 | |
| *** david-lyle has quit IRC | 18:37 | |
| *** andreww has joined #openstack-infra | 18:37 | |
| *** SumitNaiksatam has quit IRC | 18:38 | |
| *** tesseract has quit IRC | 18:39 | |
| *** mgoddard_ has joined #openstack-infra | 18:39 | |
| *** xarses_ has quit IRC | 18:41 | |
| *** felipemonteiro_ has quit IRC | 18:42 | |
| *** felipemonteiro has joined #openstack-infra | 18:42 | |
| *** mgoddard_ has left #openstack-infra | 18:43 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:43 | |
| *** alexchadin has quit IRC | 18:44 | |
| *** felipemonteiro has quit IRC | 18:47 | |
| *** signed8bit is now known as signed8bit_Zzz | 18:50 | |
| mriedem | are we ok to recheck things yet? | 18:51 |
| corvus | mriedem: we think, but we're waiting for confirmation | 18:51 |
| corvus | (confirmation takes the form of a completed tempest run) | 18:51 |
| corvus | if you're in a hurry, feel free to play the odds :) | 18:52 |
| *** e0ne has quit IRC | 18:55 | |
| fungi | http://zuul.openstack.org/stream.html?uuid=78bb7955041149d68ac623d89cef1dd0&logfile=console.log looks like it _could_ be nearing completion | 18:56 |
| *** beagles is now known as beagles|biab | 18:57 | |
| fungi | not sure if the legacy tempest jobs were affected or just the new-style | 18:57 |
| *** jamesmcarthur has quit IRC | 18:58 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:58 | |
| corvus | http://zuul.openstack.org/stream.html?uuid=8803465cd76446d6bf98c5603c08d586&logfile=console.log is finishing | 18:58 |
| *** jamesmcarthur has joined #openstack-infra | 18:58 | |
| *** jamesmcarthur has quit IRC | 18:59 | |
| *** jamesmcarthur has joined #openstack-infra | 19:00 | |
| AJaeger | fungi: the new style were affected for sure - and might be the only ones | 19:01 |
| *** dizquierdo has joined #openstack-infra | 19:01 | |
| clarkb | ya I think just new style | 19:01 |
| *** olaph1 has joined #openstack-infra | 19:02 | |
| *** olaph has quit IRC | 19:03 | |
| *** esberglu has quit IRC | 19:07 | |
| *** efoley_ has joined #openstack-infra | 19:08 | |
| *** efoley has quit IRC | 19:08 | |
| *** jamesdenton has quit IRC | 19:08 | |
| fungi | the tempest-full-py3 run just failed | 19:09 |
| *** esberglu has joined #openstack-infra | 19:09 | |
| frickler | this looks successful http://logs.openstack.org/56/522556/11/check/horizon-dsvm-tempest-plugin/181c664/job-output.txt.gz | 19:10 |
| fungi | yeah, it looks like the failure i saw wasn't post phase | 19:10 |
| fungi | was an actual tempest test failing | 19:11 |
| fungi | (two tempest tests failing, actually) | 19:11 |
| fungi | which, for our purposes, is still a success | 19:11 |
| *** esberglu_ has joined #openstack-infra | 19:12 | |
| fungi | and yeah, the one frickler found seems to have succeeded at 18:34z which is well after we restarted the executors | 19:12 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 19:13 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 19:13 |
| *** gfidente has quit IRC | 19:13 | |
| fungi | that horizon-dsvm-tempest-plugin run seems to have finished quickly because the job only runs 1 tempest test ;) | 19:14 |
| *** pblaho has quit IRC | 19:14 | |
| fungi | "Ran: 1 tests in 14.0000 sec." | 19:14 |
| *** esberglu has quit IRC | 19:14 | |
| fungi | tempest_horizon.tests.scenario.test_dashboard_basic_ops.TestDashboardBasicOps.test_basic_scenario | 19:14 |
| *** efoley_ is now known as efoley | 19:14 | |
| frickler | nova is also fine http://logs.openstack.org/22/532822/1/check/tempest-full-py3/2b48e1c/ | 19:14 |
| clarkb | sounds like it works then | 19:15 |
| pabelanger | yay | 19:15 |
| *** Swanson has joined #openstack-infra | 19:16 | |
| *** david-lyle has joined #openstack-infra | 19:17 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 19:18 |
| fungi | so... #status ok The regression stemming from one of yesterdays Zuul security fixes has been rectified, and Devstack/Tempest jobs which encountered POST_FAILURE results over the past 24 hours can safely be rechecked now | 19:18 |
| fungi | lgty? | 19:18 |
| dmsimard | lgtm | 19:18 |
| fungi | yesterday's | 19:18 |
| *** hemna_ has joined #openstack-infra | 19:18 | |
| dmsimard | indeed | 19:18 |
| *** signed8bit_Zzz is now known as signed8bit | 19:19 | |
| clarkb | yup lgtm | 19:21 |
| fungi | #status ok The regression stemming from one of yesterday's Zuul security fixes has been rectified, and Devstack/Tempest jobs which encountered POST_FAILURE results over the past 24 hours can safely be rechecked now | 19:21 |
| openstackstatus | fungi: sending ok | 19:21 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 19:24 | |
| -openstackstatus- NOTICE: The regression stemming from one of yesterday's Zuul security fixes has been rectified, and Devstack/Tempest jobs which encountered POST_FAILURE results over the past 24 hours can safely be rechecked now | 19:24 | |
| *** hongbin has quit IRC | 19:25 | |
| *** hongbin has joined #openstack-infra | 19:26 | |
| openstackstatus | fungi: finished sending ok | 19:28 |
| *** yamamoto has joined #openstack-infra | 19:29 | |
| *** frickler has quit IRC | 19:29 | |
| *** diablo_rojo_ has quit IRC | 19:30 | |
| *** diablo_rojo has joined #openstack-infra | 19:30 | |
| *** frickler has joined #openstack-infra | 19:30 | |
| *** yamamoto has quit IRC | 19:33 | |
| fungi | mriedem: ^ since you were asking recently | 19:34 |
| mriedem | yeah i saw it, thanks | 19:34 |
| *** jbadiapa has quit IRC | 19:34 | |
| * fungi needs to go find some food, then file more expense reports | 19:35 | |
| clarkb | I've grown a rather troublesome headache | 19:35 |
| clarkb | I'll probably pay attention to make sure there aren't other things I can help with like restarting executors but otherwise going to find a quiet place | 19:35 |
| fungi | construction noise and shaking hasn't subsided, i guess | 19:36 |
| clarkb | they've actually been quiet today thankfully otehrwise this would likely be worse | 19:36 |
| *** jamesmcarthur has quit IRC | 19:36 | |
| *** jamesmcarthur has joined #openstack-infra | 19:37 | |
| *** jamesmcarthur has quit IRC | 19:38 | |
| *** jamesmcarthur has joined #openstack-infra | 19:38 | |
| *** felipemonteiro_ has quit IRC | 19:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 19:40 | |
| *** beagles|biab is now known as beagles | 19:42 | |
| dmsimard | infra-root: I'll merge the ara sqlite middleware work for logs-dev now that everything is back to normal and I'll monitor things. For reference: https://review.openstack.org/#/q/topic:ara-sqlite-middleware | 19:42 |
| dmsimard | hmm, having another look at https://review.openstack.org/#/c/513874/, I'll split that out to a new role which probably makes sense and is also less prone to unexpectedly breaking the base job due to emit-ara-html not being tested. | 19:44 |
| *** olaph has joined #openstack-infra | 19:45 | |
| *** olaph1 has quit IRC | 19:48 | |
| fungi | okay, heading out to find food, back in a while | 19:50 |
| *** dave-mccowan has quit IRC | 19:53 | |
| *** dprince has quit IRC | 19:56 | |
| *** eharney has quit IRC | 20:02 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Add support for installing ARA wsgi middleware for sqlite databases https://review.openstack.org/513866 | 20:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Expand secret inheritance test https://review.openstack.org/553595 | 20:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 20:06 |
| *** lpetrut has quit IRC | 20:06 | |
| *** dsariel has quit IRC | 20:07 | |
| Swanson | If I did a recheck while zuul was in the wilderness and everything is currently queued will it be run or should I recheck again? | 20:08 |
| *** rlandy is now known as rlandy|afk | 20:09 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Expand secret inheritance test https://review.openstack.org/553595 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 20:11 |
| *** dave-mccowan has joined #openstack-infra | 20:11 | |
| corvus | Swanson: no need to recheck again | 20:12 |
| *** iyamahat_ has quit IRC | 20:13 | |
| Swanson | corvus, thanks! | 20:15 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: fixes wrong cast on location service for updatemap/image https://review.openstack.org/553539 | 20:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 20:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 20:17 |
| ianw | frickler: did you fix the backup job on zuul01? if not i'll look at today | 20:22 |
| openstackgerrit | Merged openstack-infra/system-config master: Enable and configure the ara middleware for logs-dev.o.o https://review.openstack.org/513868 | 20:25 |
| *** yamamoto has joined #openstack-infra | 20:29 | |
| *** dkehn has joined #openstack-infra | 20:31 | |
| AJaeger | config-core, if you have some spare cycles, please go over the config repos and review and approve open changes. We have again a long backlog | 20:33 |
| dmsimard | AJaeger: good to see you around, I wanted to have your opinion on something because I was conflicted | 20:33 |
| AJaeger | dmsimard: sure | 20:34 |
| dmsimard | AJaeger: I feel like https://review.openstack.org/#/c/513874/ should probably be split to another role because it has nothing to do about emitting html, what do you think ? | 20:34 |
| AJaeger | puuuh, tough one | 20:35 |
| *** yamamoto has quit IRC | 20:35 | |
| AJaeger | should we rename the role? | 20:36 |
| AJaeger | If we split it up, wouldn't there be much duplication? | 20:36 |
| *** kgiusti has left #openstack-infra | 20:37 | |
| *** LinuxJedi has joined #openstack-infra | 20:37 | |
| AJaeger | we use emit-ara-html only in project-config currently, so could rename it. | 20:37 |
| AJaeger | thinking loud: Do we need both output formats? Or could we just switch over to the new one and remove html - and rename? | 20:38 |
| AJaeger | dmsimard: no good answers from me for today ;) | 20:39 |
| dmsimard | AJaeger: if it wasn't so complicated (considering trusted, no testing and it can break everything -- including external zuuls) I would probably rename the role to something like "upload-ara-report" which would default to generating the html report but could also be toggled to upload the database instead | 20:40 |
| dmsimard | You kind of bring a good question and my brain is kind of too tired to be thinking about it right now | 20:41 |
| dmsimard | Renaming or removing roles from zuul-jobs sounds like a bad idea since we can't tell if people outside are using them | 20:41 |
| *** jamesmcarthur has quit IRC | 20:41 | |
| dmsimard | corvus: do you have any thoughts on ^ | 20:42 |
| corvus | dmsimard: yes, do it slowly with notification | 20:42 |
| dmsimard | corvus: as in, zuul-announce ? | 20:42 |
| dmsimard | corvus: I suppose if we add a deprecation notice in the role it would probably go unnoticed | 20:42 |
| corvus | dmsimard: it sounds like in this case, if anyone is using them in a job outside of zuul-jobs, it's probably an easy fix. so i don't think the deprecation period would need to be too long. maybe 4-6 weeks or something. | 20:45 |
| corvus | dmsimard: however, i don't think we need to observe even that until we've made the v3 release. | 20:46 |
| corvus | give it maybe a week? | 20:46 |
| dmsimard | corvus: that's fair. I wonder how we could make a deprecation obvious though ? | 20:46 |
| pabelanger | dmsimard: ianw: clarkb: what are the next steps for linestone cloud? | 20:47 |
| corvus | dmsimard: i think it's entirely reasonable for someone using zuul-jobs in a CD manner to follow the announce mailing list. | 20:47 |
| dmsimard | I mean, we could add in a debug warning message and then a 1 minute pause, I dunno. | 20:47 |
| corvus | dmsimard: we could use zuul_return to plumb warning messages all the way back to the reporter. | 20:48 |
| dmsimard | pabelanger: I'm waiting on https://review.openstack.org/#/q/topic:add-limestone before standing up the afs mirror | 20:48 |
| dmsimard | corvus: oh, that's clever | 20:48 |
| pabelanger | dmsimard: looks like first one has 2 +2, so I think you +3 that yourself. I've +2'd the others | 20:49 |
| pabelanger | dmsimard: you could add the nodepool clouds.yaml changes and start the image upload process too | 20:50 |
| pabelanger | in nodepool.yaml | 20:50 |
| dmsimard | pabelanger: I'm babysitting the ara middleware patches right now, I'll put limestone next on my list | 20:50 |
| pabelanger | okay, how many nodes will we get from that cloud? | 20:51 |
| dmsimard | pabelanger: I'm not sure, actually. | 20:51 |
| dmsimard | logan-: ^ how many VMs are we looking at ? | 20:51 |
| pabelanger | looks like ovh-bhs1 is having some issue | 20:52 |
| logan- | should be enough capacity there for 60 or so, should not be a big deal to double+ that as we get it up and running | 20:52 |
| *** dave-mccowan has quit IRC | 20:53 | |
| dmsimard | logan-: that's awesome, thanks a lot :) | 20:53 |
| *** iyamahat has joined #openstack-infra | 20:53 | |
| dmsimard | logan-: I'll ping you before we open the floodgates, should we settle for 50 to begin with ? | 20:53 |
| logan- | yup sounds good | 20:53 |
| dmsimard | ack | 20:53 |
| logan- | i think i need to create a flavor with the exact nodepool specs | 20:53 |
| dmsimard | logan-: indeed, there was some default flavors that didn't match the nodepool specs | 20:54 |
| logan- | yea let me do that real quick | 20:54 |
| dmsimard | thanks! | 20:54 |
| logan- | 4vcpu/8gb/80gb right? | 20:54 |
| *** hashar has joined #openstack-infra | 20:54 | |
| pabelanger | https://docs.openstack.org/infra/manual/testing.html | 20:55 |
| pabelanger | I think rax has 8VCPu | 20:55 |
| pabelanger | but depending on cloud, I think you could try with 4vcpu | 20:55 |
| logan- | ah yep https://docs.openstack.org/infra/system-config/contribute-cloud.html 8vcpu on here | 20:56 |
| logan- | that works | 20:56 |
| *** eernst has quit IRC | 20:56 | |
| logan- | it is dedicated hardware, makes no difference to me :) | 20:56 |
| pabelanger | yah, real metric is devstack so, depending on oversubscribe, i think it could be tweaked | 20:56 |
| logan- | yup can gauge load and adjust node count as we get some load on there :) | 20:57 |
| logan- | nodepool.0 flavor exists now | 20:57 |
| pabelanger | cool | 20:57 |
| pabelanger | OVH-BHS1 looks to be having message bus issues | 20:57 |
| *** sree has joined #openstack-infra | 20:58 | |
| *** trown|ruck is now known as trown|outtypewww | 20:59 | |
| *** armaan has quit IRC | 21:00 | |
| *** armaan has joined #openstack-infra | 21:00 | |
| *** armaan has quit IRC | 21:00 | |
| *** armaan has joined #openstack-infra | 21:01 | |
| *** sree has quit IRC | 21:02 | |
| *** edmondsw has quit IRC | 21:02 | |
| ianw | pabelanger: so the new xenial-based mirror-update has managed to corrupt itself in similar ways to the old version. i'm going to try the openafs client upgrade soon | 21:05 |
| logan- | dmsimard: btw ping me specs if you need any different flavor specs for the mirror or any other non-nodepool stuff | 21:06 |
| pabelanger | ianw: k | 21:07 |
| dmsimard | logan-: wfm | 21:07 |
| pabelanger | ianw: did you want to look at https://review.openstack.org/#/q/topic:add-limestone again to help bring the cloud online? | 21:07 |
| *** efoley has quit IRC | 21:08 | |
| *** myoung|rover is now known as myoung|afk | 21:12 | |
| pabelanger | corvus: looks like you have a held node for programoutput bug, can that be cleaned up? | 21:12 |
| ianw | pabelanger / dmsimard : i'm fine with it all. lmn i can make some time today to bring bits of it up if we like | 21:12 |
| pabelanger | clarkb helping slaweq debug networking issues, is also a held node. Do you need that any more? | 21:12 |
| pabelanger | mordred osc job breakage, same question about held node. Can it be deleted? | 21:13 |
| clarkb | pabelanger: I want to say the issue got sorted out and instance can go away | 21:13 |
| *** ldnunes has quit IRC | 21:15 | |
| corvus | pabelanger: yep | 21:18 |
| *** jamesmcarthur has joined #openstack-infra | 21:19 | |
| fungi | pabelanger: note that i expect mordred is probably still on a beach consuming beverages with unnecessary tiny umbrellas until tomorrow | 21:20 |
| corvus | surely they are not unecessary | 21:21 |
| fungi | he hasn't broken radio silence (according to freenode) for some 80+ hours | 21:21 |
| fungi | corvus: fair, i didn't consider the aesthetics of the beverage | 21:21 |
| corvus | fungi: without the umbrellas, they're just booze | 21:22 |
| fungi | also, they keep the booze from getting diluted in a spontaneous shower | 21:22 |
| corvus | these are problems in tropical paradise | 21:23 |
| *** r-daneel_ has joined #openstack-infra | 21:23 | |
| corvus | i like pina coladas and getting caught in the rain, but i do not like my pina coladas getting caught in the rain | 21:24 |
| slaweq | pabelanger: clarkb: yes, I already found and solve problem which we had, thx once again for help then :) | 21:24 |
| *** r-daneel has quit IRC | 21:24 | |
| *** r-daneel_ is now known as r-daneel | 21:24 | |
| slaweq | if this instance is stil there You can remove it any time | 21:24 |
| *** rossella_s has quit IRC | 21:25 | |
| fungi | corvus: got it, jimmy buffet does indeed make a fine case for necessary tiny umbrellas | 21:26 |
| *** rossella_s has joined #openstack-infra | 21:28 | |
| *** mikal_ has joined #openstack-infra | 21:30 | |
| *** yamamoto has joined #openstack-infra | 21:31 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 21:32 |
| *** mikal has quit IRC | 21:32 | |
| *** cgoncalves has quit IRC | 21:34 | |
| pabelanger | ack, cleaning up nodes, minus mordred | 21:36 |
| *** yamamoto has quit IRC | 21:36 | |
| *** efoley has joined #openstack-infra | 21:39 | |
| *** hashar has quit IRC | 21:39 | |
| *** dave-mccowan has joined #openstack-infra | 21:39 | |
| *** Goneri has quit IRC | 21:39 | |
| *** r-daneel_ has joined #openstack-infra | 21:41 | |
| *** r-daneel has quit IRC | 21:41 | |
| *** r-daneel_ is now known as r-daneel | 21:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 21:41 |
| *** esberglu_ has quit IRC | 21:41 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553619 | 21:42 |
| *** esberglu has joined #openstack-infra | 21:44 | |
| *** felipemonteiro__ has joined #openstack-infra | 21:45 | |
| *** esberglu_ has joined #openstack-infra | 21:45 | |
| *** jtomasek has quit IRC | 21:46 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553619 | 21:47 |
| *** eernst has joined #openstack-infra | 21:48 | |
| *** esberglu has quit IRC | 21:48 | |
| *** felipemonteiro_ has quit IRC | 21:49 | |
| *** esberglu_ has quit IRC | 21:50 | |
| openstackgerrit | Merged openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/551448 | 21:52 |
| adriant | just to make sure I'm not going insane, does zuul do anything clever about figuring out what tests to run? | 21:53 |
| adriant | like, I did a change that was purely docs, and it only ran pep8 | 21:53 |
| *** bobh has quit IRC | 21:53 | |
| adriant | and now rebasing another it did run the full suite of tests | 21:53 |
| *** mikal_ is now known as mikal | 21:55 | |
| *** bobh has joined #openstack-infra | 21:56 | |
| fungi | adriant: some of the standard jobs come with a set of file exclusions | 21:56 |
| adriant | fungi: oh cool :) | 21:56 |
| *** jamesmcarthur has quit IRC | 21:57 | |
| fungi | adriant: for example, the irrelevant-files pattern set in this job: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n294 | 21:58 |
| fungi | says if the change in question is only modifying files matching one or more of those patterns, then there's no need to run that particular job | 21:59 |
| adriant | fungi: yep, that would be it, the change was purely rst and in the docs folder :P | 21:59 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Refactor "emit-ara-html" into a new role: "ara-report" https://review.openstack.org/513874 | 21:59 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553621 | 21:59 |
| *** bobh has quit IRC | 21:59 | |
| dmsimard | pabelanger, AJaeger: ^ I took the "safe" route | 22:01 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553621 | 22:01 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553621 | 22:02 |
| *** danpawlik has joined #openstack-infra | 22:02 | |
| *** rcernin has joined #openstack-infra | 22:02 | |
| *** danpawlik has quit IRC | 22:02 | |
| *** rlandy|afk is now known as rlandy | 22:02 | |
| mgagne | maintenance is starting now on inap-mtl01 | 22:05 |
| fungi | thanks for the heads up, mgagne! | 22:06 |
| *** masber has joined #openstack-infra | 22:09 | |
| *** tosky has joined #openstack-infra | 22:13 | |
| *** gouthamr has quit IRC | 22:16 | |
| *** claudiub|2 has quit IRC | 22:20 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553623 | 22:24 |
| ianw | hmm, the bandersnatch mirror is now failing, with extremely unhelpful logging output; i.e. none | 22:27 |
| *** felipemonteiro__ has quit IRC | 22:27 | |
| fungi | oh? | 22:28 |
| ianw | http://paste.openstack.org/show/702291/ | 22:28 |
| fungi | yeah. i just pulled up the log myself | 22:29 |
| fungi | that one's on us. it's a traceback coming from our wrapper script | 22:29 |
| fungi | and i suspect its eating the stderr from bandersnatch | 22:30 |
| *** hemna_ has quit IRC | 22:30 | |
| *** armaan has quit IRC | 22:31 | |
| fungi | it does seem to redirect stderr=subprocess.STDOUT there, at least | 22:32 |
| fungi | so maybe bandersnatch is emitting detail on a different fd than stderr | 22:32 |
| *** yamamoto has joined #openstack-infra | 22:32 | |
| fungi | oh, or we're just not writing it out because we bail early on the non-zero exit status | 22:33 |
| fungi | we should probably wrap that subprocess.check_output() call in some exception handling so we can clean up and emit what we logged | 22:34 |
| fungi | ianw: yeah, looking more closely at /usr/local/bin/run-bandersnatch i'm pretty sure that's why we logged nothing helpful there | 22:35 |
| ianw | hmm, we should probably use the iterator approach and wait() for the process, rather than check_output? | 22:36 |
| ianw | pabelanger: and in other news | 22:37 |
| ianw | deleting and forgetting pool/main/t/texlive-base/texlive-base_2017.20180305-1_all.deb | 22:37 |
| ianw | Unable to forget unknown filekey 'pool/main/t/texlive-base/texlive-base_2017.20180305-1_all.deb'. | 22:37 |
| ianw | you ever seen that one before? | 22:37 |
| ianw | i'm starting to think maybe we shoudl just move mirror-update onto the afs rw volume server | 22:37 |
| *** yamamoto has quit IRC | 22:38 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553625 | 22:40 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-bandersnatch master: Catch subprocess output and print https://review.openstack.org/553626 | 22:41 |
| ianw | fungi: ^ simple thing, i'll put it in manually for now | 22:41 |
| openstackgerrit | Merged openstack-infra/system-config master: Install the limestone cacert https://review.openstack.org/552706 | 22:41 |
| fungi | lmk if it works when you try it there and i'll just approve | 22:41 |
| *** jamesmcarthur has joined #openstack-infra | 22:42 | |
| *** Goneri has joined #openstack-infra | 22:42 | |
| ianw | cool just running by hand now, i think it takes a bit to fail | 22:43 |
| fungi | added some notes to the change. easiest is to probably just add a raise call in the exception handler | 22:45 |
| fungi | to reraise the exception | 22:45 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553627 | 22:45 |
| ianw | indeed, otherwise you get UnboundLocalError: local variable 'output' referenced before assignment :) | 22:46 |
| openstackgerrit | Merged openstack-infra/system-config master: Add Limestone credentials to *clouds.yaml https://review.openstack.org/552707 | 22:46 |
| ianw | fungi the human python parser | 22:46 |
| fungi | i've been called worse, i suppose | 22:47 |
| ianw | fungi: well, it seems to fail for no discernible reason -> http://paste.openstack.org/show/702311/ | 22:48 |
| *** dave-mccowan has quit IRC | 22:48 | |
| ianw | maybe we're not catching all of stderr here | 22:48 |
| fungi | yeah, could be buffered and need to iterate? | 22:49 |
| clarkb | ianw: re mirror update moving I take it the current trouble is with updated packages? | 22:50 |
| fungi | ianw: do you think it's worked correctly since the server rebuild? | 22:50 |
| *** hongbin has quit IRC | 22:50 | |
| ianw | fungi: no i don't think so | 22:51 |
| fungi | did we perhaps end up with a newer bandersnatch? | 22:51 |
| fungi | i see latest release requires puthon 3.5 | 22:52 |
| ianw | the new one is python3 only | 22:52 |
| fungi | python 3.5 too | 22:52 |
| fungi | yeah | 22:52 |
| ianw | i feel like i have changes out about updating that | 22:52 |
| *** olaph1 has joined #openstack-infra | 22:52 | |
| fungi | seems like maybe we only did ensure=>present instead of ensure=>latest | 22:52 |
| fungi | checking | 22:52 |
| ianw | sigh, no --version flag | 22:53 |
| fungi | bandersnatch==1.11 installed under python2 | 22:53 |
| *** olaph has quit IRC | 22:53 | |
| *** salv-orl_ has quit IRC | 22:53 | |
| *** salv-orlando has joined #openstack-infra | 22:54 | |
| fungi | 2.1.3 is latest on pypi | 22:54 |
| fungi | i _think_ i had a backported patch of mine applied to the one on the old server which ended up only landing upstream after they decided to drop python 2 support | 22:55 |
| ianw | yeah, iirc they have merged our changes, we just never got around to updated | 22:55 |
| fungi | a patch which "fixed" an incorrect exit code | 22:55 |
| fungi | which would entirely explain what you're seeing now | 22:55 |
| *** diablo_rojo has quit IRC | 22:56 | |
| ianw | indeed ... the old server is there, just turned off | 22:56 |
| fungi | yeah, the entrypoint wrapper does a sys.exit(main()) and main() ends in "return config" | 22:57 |
| ianw | seeing as we've got this far, should we just look at updating? | 22:57 |
| fungi | so it's exiting with a ConfigParser.ConfigParser object coerced into an int, which most assuredly isn't 0 | 22:57 |
| *** signed8b_ has joined #openstack-infra | 22:58 | |
| *** signed8b_ has quit IRC | 22:58 | |
| *** salv-orlando has quit IRC | 22:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-bandersnatch master: Catch subprocess output and print https://review.openstack.org/553626 | 22:59 |
| ianw | oh, i really don't want to deal with python3-ifying the puppet right now i guess | 23:00 |
| fungi | looks like my pr was https://bitbucket.org/pypa/bandersnatch/pull-requests/23/fix-67-stop-returning-config-from-main/diff which they ignored for ages and then basically recreated in main without merging mine | 23:00 |
| *** signed8bit has quit IRC | 23:01 | |
| fungi | basically, strip the "return config" line from the end of /usr/local/lib/python2.7/dist-packages/bandersnatch/main.py | 23:01 |
| fungi | and things will work just fine | 23:01 |
| *** masber has quit IRC | 23:02 | |
| fungi | until we py3kify the configuration manglement or rebuild the server again | 23:02 |
| ianw | ok, trying again with fungi patch | 23:03 |
| fungi | my most ingenious patches involve deleting a line of code at random | 23:03 |
| fungi | or perhaps i meant ingenuous | 23:04 |
| *** adarazs has quit IRC | 23:04 | |
| mgagne | maintenance is completed in inap-mtl01. I see new nodes being built, lets see how it goes | 23:05 |
| fungi | the timeline of comments on https://bitbucket.org/pypa/bandersnatch/issues/67 makes me feel marginally better about our numerous neglected projects | 23:05 |
| *** adarazs has joined #openstack-infra | 23:06 | |
| fungi | thanks again mgagne! | 23:06 |
| ianw | yay! releasing new volume | 23:06 |
| *** ihrachys has quit IRC | 23:06 | |
| *** olaph has joined #openstack-infra | 23:08 | |
| *** olaph1 has quit IRC | 23:09 | |
| *** diablo_rojo has joined #openstack-infra | 23:10 | |
| mgagne | are there logs for nodepool? just want to make sure there are no problem | 23:11 |
| *** tellesnobrega has quit IRC | 23:11 | |
| mgagne | because there are a bunch of VMs in BUILD state being deleted | 23:11 |
| *** dhill_ has quit IRC | 23:11 | |
| mgagne | maybe a timeout due to the number of VMs being created at the same time? | 23:11 |
| mgagne | how I perceive Nodepool right now =) https://i.imgur.com/jZVbrGU.gifv | 23:12 |
| *** olaph1 has joined #openstack-infra | 23:16 | |
| fungi | it's too bad we ended u with liberty instead of lemming for our l release | 23:16 |
| fungi | i'll skim the logs | 23:16 |
| *** olaph has quit IRC | 23:17 | |
| fungi | my guess, like yours, is that we ended up making too many simultaneous boot calls, many of which didn't respond in a timely fashion, so we issued deletes for those and started to boot replacements | 23:18 |
| corvus | that's actually our original design document for nodepool | 23:18 |
| fungi | hah | 23:18 |
| mgagne | oh well | 23:18 |
| fungi | looks like nl03 is the one i want | 23:19 |
| fungi | that's nodepool lemming 03 for those who didn't know | 23:19 |
| fungi | yeah, logging quite a few "Timeout waiting for server..." | 23:20 |
| mgagne | ok so it "should" solve by itself over time | 23:20 |
| fungi | on deletion calls but also | 23:20 |
| fungi | "Timeout waiting for the server to come up." | 23:20 |
| fungi | so, yes, i expect it will settle down to a steady state here momentarily | 23:21 |
| fungi | looks like it's mostly just retried server delete calls it's complaining about timeouts for now | 23:25 |
| corvus | it looks like most of the change is building -> deleting... are the deletions taking a while? | 23:25 |
| mgagne | yea, checking that | 23:25 |
| *** yamamoto has joined #openstack-infra | 23:27 | |
| *** hamzy__ has quit IRC | 23:29 | |
| *** HeOS has quit IRC | 23:33 | |
| fungi | still seeing plenty of deletion timeouts and the occasional boot timeout | 23:34 |
| *** aeng has joined #openstack-infra | 23:34 | |
| *** rcernin has quit IRC | 23:39 | |
| *** HeOS has joined #openstack-infra | 23:41 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Update neutron-dynamic-routing grafana https://review.openstack.org/553635 | 23:44 |
| *** tellesnobrega has joined #openstack-infra | 23:44 | |
| pabelanger | ianw: you should be able to manually forget entries. I've done that in the past. | 23:45 |
| ianw | pabelanger: yeah, it won't forget these for some reason. i've re-downloaded the files, _detected them, and trying again | 23:49 |
| *** olaph has joined #openstack-infra | 23:51 | |
| *** olaph1 has quit IRC | 23:51 | |
| *** salv-orlando has joined #openstack-infra | 23:54 | |
| *** olaph1 has joined #openstack-infra | 23:55 | |
| pabelanger | ianw: maybe copy the ro database over and redetect? | 23:55 |
| ianw | really trying to avoid that, it's 6+ hours worth :/ | 23:56 |
| pabelanger | yah, I think it might be worth to split 1 release per database in reprepro, to make recover time faster | 23:57 |
| *** olaph has quit IRC | 23:57 | |
| *** olaph1 is now known as olaph | 23:58 | |
| *** david-lyle has quit IRC | 23:59 | |
| *** salv-orlando has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!