| *** armax has joined #openstack-infra | 00:01 | |
| *** aaronsheffield has quit IRC | 00:01 | |
| *** rcernin has joined #openstack-infra | 00:08 | |
| *** whoami-rajat has joined #openstack-infra | 00:09 | |
| *** igordc has joined #openstack-infra | 00:09 | |
| *** markvoelker has joined #openstack-infra | 00:10 | |
| dmsimard | cloudnull: nested ansible | 00:13 |
|---|---|---|
| *** eernst_ has joined #openstack-infra | 00:14 | |
| *** markvoelker has quit IRC | 00:16 | |
| *** mattw4 has quit IRC | 00:16 | |
| *** eernst_ has quit IRC | 00:18 | |
| *** markvoelker has joined #openstack-infra | 00:21 | |
| *** fdegir has quit IRC | 00:23 | |
| *** georgk has quit IRC | 00:23 | |
| *** georgk has joined #openstack-infra | 00:24 | |
| *** fdegir has joined #openstack-infra | 00:24 | |
| cloudnull | dmsimard yup, that's what I ended up doing | 00:31 |
| ianw | oh interesting, i can replicate the arm64 out-of-disk during build issues in the gate | 00:32 |
| ianw | sorryt, that was for -dib | 00:32 |
| *** trident has quit IRC | 00:38 | |
| *** gyee has quit IRC | 00:48 | |
| *** ekultails has quit IRC | 00:48 | |
| *** rcernin has quit IRC | 00:58 | |
| *** rascasoft has quit IRC | 01:05 | |
| *** spsurya has joined #openstack-infra | 01:05 | |
| *** rascasoft has joined #openstack-infra | 01:06 | |
| *** auristor has quit IRC | 01:11 | |
| *** rcernin has joined #openstack-infra | 01:14 | |
| *** hamzy has joined #openstack-infra | 01:17 | |
| *** markvoelker has quit IRC | 01:23 | |
| *** markvoelker has joined #openstack-infra | 01:23 | |
| *** auristor has joined #openstack-infra | 01:24 | |
| *** markvoelker has quit IRC | 01:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:33 | |
| *** bhavikdbavishi has quit IRC | 01:38 | |
| *** rfarr has joined #openstack-infra | 01:41 | |
| *** rfarr has quit IRC | 01:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:50 | |
| *** rfolco has quit IRC | 01:52 | |
| *** markvoelker has joined #openstack-infra | 01:56 | |
| *** rh-jelabarre has joined #openstack-infra | 01:58 | |
| *** xarses has joined #openstack-infra | 02:00 | |
| *** markvoelker has quit IRC | 02:00 | |
| *** bhavikdbavishi has quit IRC | 02:05 | |
| *** rascasoft has quit IRC | 02:06 | |
| *** whoami-rajat has quit IRC | 02:28 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] arm64 functional testing https://review.opendev.org/676111 | 02:31 |
| *** yamamoto has quit IRC | 02:33 | |
| *** xinranwang has joined #openstack-infra | 02:42 | |
| *** bobh has joined #openstack-infra | 02:42 | |
| *** bobh has quit IRC | 02:44 | |
| *** bobh has joined #openstack-infra | 02:56 | |
| *** bobh has quit IRC | 02:57 | |
| *** yamamoto has joined #openstack-infra | 03:05 | |
| *** ccamacho has joined #openstack-infra | 03:06 | |
| *** strigazi has quit IRC | 03:09 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Don't show all elements found https://review.opendev.org/676312 | 03:09 |
| *** strigazi has joined #openstack-infra | 03:10 | |
| *** sshnaidm|afk has quit IRC | 03:10 | |
| *** yamamoto has quit IRC | 03:13 | |
| *** whoami-rajat has joined #openstack-infra | 03:20 | |
| *** bobh has joined #openstack-infra | 03:21 | |
| *** sshnaidm|afk has joined #openstack-infra | 03:23 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:27 | |
| *** bobh has quit IRC | 03:27 | |
| *** psachin has joined #openstack-infra | 03:33 | |
| *** ykarel|away has joined #openstack-infra | 03:38 | |
| *** armax has quit IRC | 03:41 | |
| *** ramishra has joined #openstack-infra | 03:46 | |
| *** udesale has joined #openstack-infra | 03:54 | |
| *** bobh has joined #openstack-infra | 03:56 | |
| *** bobh has quit IRC | 03:58 | |
| *** markvoelker has joined #openstack-infra | 04:01 | |
| *** markvoelker has quit IRC | 04:05 | |
| *** yamamoto has quit IRC | 04:08 | |
| *** ykarel|away is now known as ykarel | 04:11 | |
| *** raukadah is now known as chkumar|ruck | 04:33 | |
| *** rh-jelabarre has quit IRC | 04:42 | |
| *** markvoelker has joined #openstack-infra | 04:52 | |
| *** markvoelker has quit IRC | 04:57 | |
| *** igordc has quit IRC | 05:02 | |
| *** yamamoto has joined #openstack-infra | 05:22 | |
| *** ykarel is now known as ykarel|afk | 05:26 | |
| *** yamamoto has quit IRC | 05:27 | |
| *** kopecmartin|off is now known as kopecmartin | 05:36 | |
| *** odicha has joined #openstack-infra | 05:38 | |
| openstackgerrit | Sundar Nadathur proposed openstack/project-config master: End gating for os-acc as that project is about to be retired. https://review.opendev.org/676331 | 05:52 |
| *** dpawlik has joined #openstack-infra | 06:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add arm64 based functional test https://review.opendev.org/676111 | 06:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: block-device-efi : expand disk size calculation https://review.opendev.org/676354 | 06:14 |
| *** rcernin has quit IRC | 06:16 | |
| ianw | logan-: as someone who has looked at similar things, any thoughts on ^ welcome | 06:16 |
| *** dpawlik has quit IRC | 06:20 | |
| *** jtomasek has joined #openstack-infra | 06:21 | |
| *** dpawlik has joined #openstack-infra | 06:23 | |
| *** gfidente has joined #openstack-infra | 06:24 | |
| *** gfidente has quit IRC | 06:30 | |
| *** rcernin has joined #openstack-infra | 06:31 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add arm64 based functional test https://review.opendev.org/676111 | 06:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: openstack-ci-mirrors: use ubuntu-ports variable https://review.opendev.org/676358 | 06:49 |
| *** slaweq has joined #openstack-infra | 06:52 | |
| *** trident has joined #openstack-infra | 07:01 | |
| *** ccamacho has quit IRC | 07:10 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Don't show all elements found https://review.opendev.org/676312 | 07:10 |
| *** tesseract has joined #openstack-infra | 07:10 | |
| *** odicha has quit IRC | 07:12 | |
| *** odicha has joined #openstack-infra | 07:15 | |
| *** ykarel|afk is now known as ykarel | 07:19 | |
| *** kaisers has quit IRC | 07:21 | |
| *** xinranwang has quit IRC | 07:21 | |
| *** e0ne has joined #openstack-infra | 07:22 | |
| *** janki has joined #openstack-infra | 07:23 | |
| *** kaisers has joined #openstack-infra | 07:23 | |
| *** ykarel is now known as ykarel|afk | 07:33 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** yamamoto has joined #openstack-infra | 07:43 | |
| *** apetrich has joined #openstack-infra | 07:45 | |
| *** yamamoto has quit IRC | 07:46 | |
| *** ykarel|afk has quit IRC | 07:47 | |
| *** gfidente has joined #openstack-infra | 07:53 | |
| *** rcernin has quit IRC | 07:56 | |
| *** lucasagomes has joined #openstack-infra | 08:02 | |
| *** ykarel|afk has joined #openstack-infra | 08:02 | |
| *** elod_off is now known as elod | 08:03 | |
| *** ykarel|afk is now known as ykarel | 08:04 | |
| *** yamamoto has joined #openstack-infra | 08:05 | |
| *** markvoelker has joined #openstack-infra | 08:07 | |
| *** tkajinam has quit IRC | 08:11 | |
| *** jbadiapa has joined #openstack-infra | 08:12 | |
| *** pkopec has joined #openstack-infra | 08:15 | |
| *** markvoelker has quit IRC | 08:15 | |
| *** rpittau|afk is now known as rpittau | 08:18 | |
| *** rcernin has joined #openstack-infra | 08:23 | |
| *** derekh has joined #openstack-infra | 08:29 | |
| *** rcernin has quit IRC | 08:29 | |
| *** yamamoto has quit IRC | 08:34 | |
| *** yamamoto has joined #openstack-infra | 08:36 | |
| *** kjackal has joined #openstack-infra | 08:36 | |
| *** rcernin has joined #openstack-infra | 08:38 | |
| *** markvoelker has joined #openstack-infra | 08:41 | |
| *** janki has quit IRC | 08:44 | |
| *** markvoelker has quit IRC | 08:45 | |
| *** factor has quit IRC | 08:58 | |
| *** icarusfactor has joined #openstack-infra | 08:58 | |
| *** gfidente has quit IRC | 09:00 | |
| *** ykarel is now known as ykarel|afk | 09:02 | |
| *** jbadiapa has quit IRC | 09:03 | |
| *** rcernin has quit IRC | 09:04 | |
| *** janki has joined #openstack-infra | 09:06 | |
| *** markvoelker has joined #openstack-infra | 09:10 | |
| *** markvoelker has quit IRC | 09:15 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:23 | |
| *** tdasilva has joined #openstack-infra | 09:25 | |
| *** ykarel|afk is now known as ykarel | 09:29 | |
| *** janki has quit IRC | 09:40 | |
| *** ociuhandu has joined #openstack-infra | 09:49 | |
| *** icarusfactor has quit IRC | 09:51 | |
| *** yamamoto has quit IRC | 09:51 | |
| *** icarusfactor has joined #openstack-infra | 09:51 | |
| *** apetrich has quit IRC | 10:03 | |
| *** kjackal has quit IRC | 10:04 | |
| *** markvoelker has joined #openstack-infra | 10:11 | |
| *** ociuhandu has quit IRC | 10:14 | |
| *** ociuhandu has joined #openstack-infra | 10:15 | |
| *** markvoelker has quit IRC | 10:15 | |
| *** verdurin has joined #openstack-infra | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:24 | |
| *** yamamoto has quit IRC | 10:25 | |
| *** yamamoto has joined #openstack-infra | 10:26 | |
| *** gfidente has joined #openstack-infra | 10:27 | |
| frickler | infra-root: not sure whether that's intended, but for jobs like openstack-tox-docs the result url used to point directly to the html/ directory, now the log url from the zuul build page points to the base url like for any other job | 10:32 |
| *** bnemec has quit IRC | 10:34 | |
| *** bnemec has joined #openstack-infra | 10:37 | |
| *** dpawlik has quit IRC | 10:41 | |
| *** bnemec has quit IRC | 10:45 | |
| *** kjackal has joined #openstack-infra | 10:47 | |
| mordred | frickler: it is, but it looks lke we're missing a returned artifact to put the link to the built html on the main page ... one sec, let me get an example | 10:48 |
| *** eharney has quit IRC | 10:48 | |
| *** bnemec has joined #openstack-infra | 10:49 | |
| *** priteau has joined #openstack-infra | 10:50 | |
| mordred | frickler: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-javascript-output/tasks/main.yaml#L144-L150 causes the "Site preview" link to exist on https://zuul.opendev.org/t/zuul/build/07e89d87dd6249da84b3223ec7a4f1ab | 10:52 |
| *** bnemec has quit IRC | 11:04 | |
| *** udesale has quit IRC | 11:05 | |
| *** dpawlik has joined #openstack-infra | 11:08 | |
| *** bnemec has joined #openstack-infra | 11:09 | |
| *** udesale has joined #openstack-infra | 11:12 | |
| *** bnemec has quit IRC | 11:13 | |
| frickler | mordred: hmm, so all the success-url statements are essentially void now and we'd need a mass change to replace them with zuul_return artifacts? https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L413 | 11:14 |
| *** udesale has quit IRC | 11:16 | |
| frickler | or can we get zuul to provide an automatic artifact for the success-url? | 11:18 |
| frickler | also, do we have an example of the swift logging setup visible somewhere already? | 11:20 |
| *** yamamoto has quit IRC | 11:21 | |
| *** markvoelker has joined #openstack-infra | 11:21 | |
| *** markvoelker has quit IRC | 11:26 | |
| *** bnemec has joined #openstack-infra | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:27 | |
| *** markvoelker has joined #openstack-infra | 11:31 | |
| *** bnemec has quit IRC | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** markvoelker has quit IRC | 11:36 | |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul master: Revert "Expose date time as facts" https://review.opendev.org/676393 | 11:37 |
| *** pkopec has quit IRC | 11:38 | |
| *** jaosorior has joined #openstack-infra | 11:38 | |
| *** bnemec has joined #openstack-infra | 11:38 | |
| *** factor__ has joined #openstack-infra | 11:41 | |
| *** icarusfactor has quit IRC | 11:43 | |
| *** bnemec has quit IRC | 11:45 | |
| *** pkopec has joined #openstack-infra | 11:47 | |
| *** bnemec has joined #openstack-infra | 11:48 | |
| *** bnemec has quit IRC | 11:55 | |
| *** ekultails has joined #openstack-infra | 11:56 | |
| *** bnemec has joined #openstack-infra | 11:59 | |
| *** factor__ has quit IRC | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:02 | |
| *** rh-jelabarre has joined #openstack-infra | 12:03 | |
| *** bnemec has quit IRC | 12:04 | |
| *** bnemec has joined #openstack-infra | 12:10 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** yamamoto has quit IRC | 12:18 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** rlandy has joined #openstack-infra | 12:22 | |
| *** rlandy is now known as rlandy|rover | 12:22 | |
| *** rfolco has joined #openstack-infra | 12:26 | |
| *** bnemec has quit IRC | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| *** bnemec has joined #openstack-infra | 12:33 | |
| *** rcernin has joined #openstack-infra | 12:34 | |
| *** alemgeta has joined #openstack-infra | 12:38 | |
| *** pkopec has quit IRC | 12:39 | |
| *** rcernin has quit IRC | 12:40 | |
| openstackgerrit | Michal Nasiadka proposed openstack/project-config master: Add ceph/ceph-ansible to untrusted github projects https://review.opendev.org/676402 | 12:40 |
| *** bnemec has quit IRC | 12:41 | |
| *** bnemec has joined #openstack-infra | 12:44 | |
| *** priteau has quit IRC | 12:48 | |
| *** bnemec has quit IRC | 12:49 | |
| *** apetrich has joined #openstack-infra | 12:50 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: resolve build root real path https://review.opendev.org/676404 | 12:50 |
| *** bnemec has joined #openstack-infra | 12:54 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint https://review.opendev.org/621344 | 12:55 |
| AJaeger_ | frickler, thanks for bringing this up. I was looking at docs earlier this morning and also struggled. mordred, corvus, this is now far worse for docs and needs changing. | 12:55 |
| *** jaosorior has quit IRC | 12:56 | |
| *** AJaeger_ is now known as AJaeger | 12:57 | |
| *** pkopec has joined #openstack-infra | 12:58 | |
| *** mriedem has joined #openstack-infra | 13:00 | |
| *** aaronsheffield has joined #openstack-infra | 13:02 | |
| *** jcoufal has joined #openstack-infra | 13:04 | |
| *** guoqiao has quit IRC | 13:05 | |
| aspiers | AJaeger: FWIW I actually prefer it this way, even though it requires an extra couple of clicks. IIUC the old way made it hard to find the rest of the build info | 13:06 |
| aspiers | although ideally Gerrit would provide links to both | 13:07 |
| *** bnemec has quit IRC | 13:07 | |
| *** eharney has joined #openstack-infra | 13:07 | |
| aspiers | I guess that's probably hoping for too much though | 13:08 |
| *** bnemec has joined #openstack-infra | 13:10 | |
| *** jeliu_ has joined #openstack-infra | 13:11 | |
| AJaeger | aspiers: I'm fine having it this way - but there needs to be a pointer to make prebuild docs visible. Right now they are far to difficult to find | 13:12 |
| aspiers | agreed | 13:12 |
| aspiers | can Gerrit support multiple links? | 13:12 |
| aspiers | how flexible is its UI for CI results? | 13:12 |
| *** ykarel is now known as ykarel|away | 13:15 | |
| *** bnemec has quit IRC | 13:16 | |
| *** beekneemech has joined #openstack-infra | 13:16 | |
| frickler | AJaeger: yes, it was in fact your devstack patch where I noticed this. "html/" is still pretty obvious to find from the log url, but things like "tox/reports.html" from openstack-tox-molecule are pretty difficult to discover for a casual user IMO | 13:16 |
| AJaeger | frickler: tox/reports.html is for coverage as well. I consider even html difficult to find. For those looking the first time at such a job, it's not obvious that there is content to look at. I love to be able to find logs easily now ;) | 13:22 |
| AJaeger | frickler: btw thanks for reviewing the devstack change! | 13:22 |
| AJaeger | wait - for coverage, it might be "cover/index.html" | 13:23 |
| *** beekneemech has quit IRC | 13:33 | |
| *** bnemec has joined #openstack-infra | 13:39 | |
| *** bnemec has quit IRC | 13:47 | |
| *** pkopec has quit IRC | 13:50 | |
| *** bnemec has joined #openstack-infra | 13:51 | |
| *** jaosorior has joined #openstack-infra | 13:51 | |
| *** bnemec has quit IRC | 13:55 | |
| *** bnemec has joined #openstack-infra | 13:58 | |
| *** odicha has quit IRC | 14:01 | |
| *** sthussey has joined #openstack-infra | 14:04 | |
| *** noorul has joined #openstack-infra | 14:04 | |
| *** jaosorior has quit IRC | 14:16 | |
| corvus | okay, so has anyone actually identified where we're missing artifact links? | 14:19 |
| AJaeger | corvus: I guess every job that uses "success-url" | 14:20 |
| corvus | i see that 4 hours ago, mordred said we are. where are the changes to add them? or at least, where are the links to the jobs/roles where they need to be added? | 14:20 |
| *** dpawlik has quit IRC | 14:20 | |
| *** ociuhandu has joined #openstack-infra | 14:21 | |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: projects: add openstack/ansible-plugin-container-connection https://review.opendev.org/676421 | 14:25 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: projects: add openstack/ansible-role-uwsgi to zuul https://review.opendev.org/676422 | 14:25 |
| *** ociuhandu has quit IRC | 14:26 | |
| *** jcoufal_ has joined #openstack-infra | 14:29 | |
| mordred | corvus: I haven't made any patches for that yet, but I can get started on it | 14:29 |
| corvus | i'm looking into what's missing for the docs jobs | 14:30 |
| AJaeger | corvus, I'm currently looking at openstack-manuals | 14:30 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: WIP: docs: add test jobs howto https://review.opendev.org/676424 | 14:30 |
| AJaeger | mordred, corvus, https://review.opendev.org/676423 is my change | 14:30 |
| mnaser | AJaeger: btw we missed in the project creation patch adding uwsgi to zuul | 14:30 |
| mnaser | see https://review.opendev.org/#/c/676422/ | 14:30 |
| mnaser | ah ok | 14:30 |
| mnaser | you already got it :P | 14:30 |
| AJaeger | mnaser: yeah ;( thanks for catching it ;) | 14:30 |
| mnaser | no worries! | 14:30 |
| *** jcoufal has quit IRC | 14:31 | |
| corvus | AJaeger: the opendev tox jobs use fetch-sphinx-tarball in the post playbook which returns the artifact. this is a new set of jobs which can be used in promote (so we don't rebuild docs in post). at some point someone should look into porting the openstack docs jobs over to inherit from these, but i won't have time for that for a while. | 14:33 |
| *** ijw has joined #openstack-infra | 14:33 | |
| corvus | AJaeger: this explains why zuul and opendev jobs are getting the links | 14:33 |
| corvus | i don't believe any of the roles in tox-docs return the artifact | 14:34 |
| AJaeger | corvus: yes, switching to promote jobs would be great... | 14:34 |
| corvus | i'm looking into whether we can add it to a role that's it tox-docs for a quicker fix | 14:35 |
| corvus | AJaeger, mordred: but see my comment on 676423 | 14:35 |
| corvus | https://zuul.opendev.org/t/zuul/build/b0292765552347069309bd3c9bf0985a is a zuul docs build with that system | 14:37 |
| AJaeger | corvus: tox-docs publishes to html/ - while the job I change to publish-docs, so I would need to overwrite that in the parent | 14:37 |
| corvus | (note, we also get a docs archive for free) | 14:37 |
| AJaeger | corvus: nice. Let me play a bit with openstack-manuals since that's something easy to experiment with for me and has limited impact | 14:39 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: WIP: docs: add test jobs howto https://review.opendev.org/676424 | 14:39 |
| mordred | AJaeger: I can help out with a port if you need it - but I'll let you play for a bit | 14:40 |
| AJaeger | mordred: I'll ask for help later ;) | 14:41 |
| AJaeger | thanks, mordred | 14:41 |
| corvus | AJaeger: the manuals job and the regular tox-docs jobs both use fetch-sphinx-output; it *feels* like there should be a way to have the role return the correct artifact url, but i haven't quite figured it out yet | 14:41 |
| mordred | corvus: for tox-docs - can't we just add fetch-sphinx-tarball? | 14:43 |
| corvus | mordred: it's a replacement for fetch-sphinx-output | 14:44 |
| corvus | (the tarball comes into play because that's now an artifact that can be fetched and promoted) | 14:45 |
| mordred | corvus: yeah - sorry - I meant can't we just replace fetch-sphinx-output with fetch-sphinx-tarball in the normal tox-docs job? | 14:45 |
| corvus | and to avoid copying it twice, we copy the tarball to the executor, then extract it to create the preview site | 14:45 |
| corvus | mordred: i don't know, that's a good question to answer | 14:46 |
| mordred | how about I try it and see - I think zuul will tell us | 14:46 |
| *** noorul has quit IRC | 14:46 | |
| AJaeger | corvus: yeah, we need to change that - for releasenotes, api-ref, api-guide as well. They all use a different output dir | 14:46 |
| corvus | mordred: you'll need a depends-on since zuul-jobs uses the opendev jobs | 14:47 |
| AJaeger | corvus: sphinx_output_src is set for the manuals job | 14:47 |
| mordred | corvus: yah | 14:47 |
| corvus | AJaeger: is sphinx_output_src set for the other repos? | 14:48 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Update zuul-changes.py to python3 only https://review.opendev.org/676429 | 14:49 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Switch to fetch-sphinx-tarball for tox-docs https://review.opendev.org/676430 | 14:49 |
| *** pkopec has joined #openstack-infra | 14:49 | |
| AJaeger | corvus: yes, I think sphinx_output_src is the variable you need. | 14:50 |
| *** ramishra has quit IRC | 14:50 | |
| AJaeger | it is used for releasenotes jobs as well | 14:50 |
| *** priteau has joined #openstack-infra | 14:50 | |
| AJaeger | wait, we have sphinx_build_dir as well - for api-ref | 14:50 |
| corvus | AJaeger: yeah, unfortunately, it has zuul.project.src_dir in it, so we'll still probably need changes to the repos using it, but i think we might be able to build something on that | 14:50 |
| AJaeger | sorry, have some meetings now and can't look further into it for the next few hours... | 14:51 |
| mordred | corvus, AJaeger: https://review.opendev.org/676432 DNM Test patch for sphinx job - depends-on patch for the tox-docs change | 14:52 |
| *** armax has joined #openstack-infra | 14:56 | |
| *** ykarel|away has quit IRC | 14:56 | |
| *** ociuhandu has joined #openstack-infra | 14:57 | |
| *** pkopec has quit IRC | 14:59 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return preview artifact in fetch-sphinx-output https://review.opendev.org/676437 | 14:59 |
| *** liuyulong has quit IRC | 14:59 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: WIP: docs: add test jobs howto https://review.opendev.org/676424 | 15:00 |
| *** pkopec has joined #openstack-infra | 15:01 | |
| mordred | corvus: that looks good | 15:01 |
| corvus | remote: https://review.opendev.org/676438 Update fetch-sphinx-output parameters | 15:02 |
| corvus | mordred, AJaeger: ^ i think that's what we would need to do for the jobs which don't use "html/" as the preview site location | 15:03 |
| AJaeger | thanks, corvus - will review later | 15:03 |
| *** chkumar|ruck is now known as raukadah | 15:05 | |
| *** pkopec has quit IRC | 15:06 | |
| *** pkopec has joined #openstack-infra | 15:07 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Add appending yaml log plugin https://review.opendev.org/623256 | 15:08 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Process yaml log files if they exist https://review.opendev.org/676246 | 15:08 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Write errors from ansible execution into yaml log https://review.opendev.org/676250 | 15:08 |
| openstackgerrit | Merged zuul/zuul master: executor: resolve build root real path https://review.opendev.org/676404 | 15:11 |
| mordred | corvus: I'm free and available all day if we want to flip the swift switch fwiw | 15:11 |
| *** ociuhandu has quit IRC | 15:11 | |
| *** ociuhandu has joined #openstack-infra | 15:12 | |
| *** jcoufal_ has quit IRC | 15:12 | |
| *** ijw_ has joined #openstack-infra | 15:12 | |
| *** jcoufal has joined #openstack-infra | 15:13 | |
| *** slaweq has quit IRC | 15:14 | |
| corvus | mordred: cool, i have an appointment around 1800 -- will you be around then? | 15:14 |
| *** ijw has quit IRC | 15:14 | |
| corvus | call it 1800-2000 with lunch in there | 15:15 |
| *** slaweq has joined #openstack-infra | 15:19 | |
| *** ianychoi has joined #openstack-infra | 15:20 | |
| *** dave-mccowan has joined #openstack-infra | 15:21 | |
| *** ykarel|away has joined #openstack-infra | 15:21 | |
| *** slaweq has quit IRC | 15:24 | |
| mordred | corvus: yup | 15:24 |
| corvus | mordred: cool, i'll hit the +W | 15:24 |
| corvus | mordred: oh, actually, you hit the +W since it needs another +2 :) https://review.opendev.org/676281 | 15:25 |
| *** kjackal has quit IRC | 15:26 | |
| *** dave-mccowan has quit IRC | 15:26 | |
| mordred | corvus: done! (I thought I'd added the second +2 yesterday) | 15:28 |
| openstackgerrit | Merged opendev/base-jobs master: Store logs in swift https://review.opendev.org/676281 | 15:36 |
| clarkb | exciting | 15:37 |
| clarkb | I'll be around for about ~50 minutes and can help keep an eye on ^ | 15:37 |
| clarkb | (it should apply as soon as the zuul config loads after the merge which means it should already be done for jobs that start nowish) | 15:37 |
| *** ijw has joined #openstack-infra | 15:39 | |
| mordred | timburke: ^^ enjou | 15:39 |
| mordred | enjoy | 15:39 |
| *** ijw_ has quit IRC | 15:42 | |
| clarkb | change 643668,11 in openstack's check queue has a successful job with url pointing at swift | 15:42 |
| clarkb | and I can browse logs for it at https://cf0f5f3cf3873fd8b4b8-b0f74dea23666e59f1fd76c410995d5c.ssl.cf1.rackcdn.com/643668/11/check/test-release-openstack/439acb1/ | 15:43 |
| mordred | woot! | 15:43 |
| tdasilva | mordred: out of curiosity, what "other" features of swift are you using for this project to store logs (e.g., object expiration, tempurl, staticweb)? | 15:43 |
| clarkb | tdasilva: I think just object expiration | 15:45 |
| mordred | tdasilva: and form post upload | 15:45 |
| clarkb | and we don't static web because we want zuul to be the canonical rendering | 15:45 |
| *** Pavan has joined #openstack-infra | 15:46 | |
| clarkb | though maybe we use it for the intermediate urls? | 15:46 |
| *** ijw_ has joined #openstack-infra | 15:46 | |
| Pavan | hi I tried to setup ec2-api | 15:46 |
| Pavan | getting following error "No project with a name or ID of 'service' exists." | 15:46 |
| tdasilva | clarkb, mordred: very cool, thanks! | 15:47 |
| clarkb | Pavan: we run the developer infrastructure for openstack (code review, CI tools, etc). Unfortunately that means we aren't a great resource for running openstack in specific configurations. You might have better luck with the openstack-discuss mailing list or #openstack on freenode | 15:47 |
| mordred | tdasilva: we're very excited about getting off of our static log server :) | 15:48 |
| *** diablo_rojo has joined #openstack-infra | 15:48 | |
| mordred | I mean, having 12 1TB cinder volumes stitched into a single LV with a filesystem across it is fun and all ... | 15:48 |
| *** jcoufal has quit IRC | 15:48 | |
| tdasilva | but infinite storage is more fun? :) | 15:49 |
| mordred | tdasilva: it's also worth noting that we're using 7 regions of swift across 3 different public clouds - so yay for multi-cloud | 15:49 |
| zbr|flu | clarkb: mordred: are we deliberately using an out of support ansible version? https://opendev.org/zuul/zuul-jobs/src/branch/master/test-requirements.txt#L15 | 15:49 |
| *** ijw has quit IRC | 15:49 | |
| clarkb | zbr|flu: I believe that is only used by the linter jobs and not for explicitly running ansible anywhere | 15:50 |
| clarkb | zbr|flu: possible it was just missed | 15:50 |
| zbr|flu | clarkb: what value do you want me to put there? | 15:50 |
| clarkb | zbr|flu: maybe we should use the default zuul ansible version which is currently. 2.7 ? | 15:51 |
| zbr|flu | can we use a constraints approach in this place? | 15:51 |
| *** mattw4 has joined #openstack-infra | 15:52 | |
| mordred | we should actually circle back around and think about this - because it was done when zuul only supported one version of ansible | 15:52 |
| clarkb | mordred: well and lint is only gonna run with one version of ansible but it doesn't even use the ansible parser aiui so it doesn't matter that much | 15:53 |
| tdasilva | mordred: and what swift client are you using? | 15:53 |
| mordred | we should almost certainly be testing against the set of supported ansible versions, rather than just trailing the zuul version | 15:53 |
| clarkb | the current value is probably actually fine | 15:53 |
| mordred | clarkb: wow | 15:53 |
| mordred | tdasilva: openstacksdk | 15:53 |
| mnaser | :O | 15:53 |
| mnaser | officially merged swift logs | 15:53 |
| mordred | mnaser: heck yes | 15:53 |
| mnaser | btw something i've noticed is with the whole promote docs thing + gerrit is that the promote thing overrides the latest build results in gerrit | 15:54 |
| mnaser | aka https://review.opendev.org/#/c/676281/2 only shows promote up top (other than in comments) | 15:54 |
| mordred | tdasilva: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-swift is the role that uploads things, and https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-swift/library/zuul_swift_upload.py is the python behind it | 15:54 |
| *** rpittau is now known as rpittau|afk | 15:55 | |
| zbr|flu | i am fine to add dual testing: min and current ansible, but forcing something that is totally out of support does not make sense to me. | 15:56 |
| Shrews | mordred: rax is part of our swift solution, yes? | 15:56 |
| mordred | Shrews: yup | 15:56 |
| Shrews | mordred: we should probably make sure we are using the newest sdk about to be released | 15:56 |
| *** tesseract has quit IRC | 15:56 | |
| Shrews | i suspect more object leaking, otherwise | 15:56 |
| zbr|flu | i can give an example from molecule, where we currently support ansible N-2, following https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html | 15:56 |
| Shrews | and by a larger facter, with all the logs | 15:57 |
| mordred | Shrews: well - luckily we're not deleting anything ourselves | 15:57 |
| mordred | Shrews, tdasilva: https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L66-L71 is the list of cloud/regions of swift | 15:57 |
| Shrews | *nod* | 15:59 |
| mordred | Shrews: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-swift/library/zuul_swift_upload.py#L577-L578 | 16:00 |
| mordred | Shrews: delete-after is our friend :) | 16:00 |
| Shrews | mordred: i wasn't aware that header was a thing. cool | 16:01 |
| mordred | Shrews: ooh. you know ... | 16:02 |
| * Shrews thinking same thing possibly | 16:02 | |
| mordred | Shrews: we should totally set it for the objects we upload for glance import | 16:02 |
| Shrews | yup :) | 16:02 |
| mordred | ++ | 16:02 |
| mordred | set it to a day or so - because we don't need it once import is done | 16:03 |
| Shrews | yeah. i'll work on that | 16:03 |
| mordred | woot | 16:03 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Correct version of ansible from test-requirements.txt https://review.opendev.org/676448 | 16:06 |
| *** yamamoto has quit IRC | 16:09 | |
| zbr|flu | sadly the zuul/lib/ansible-config.conf seems to be the only place mentioning ansible versions and it cannot be used as a constraints file. | 16:09 |
| *** yamamoto has joined #openstack-infra | 16:10 | |
| *** yamamoto has quit IRC | 16:10 | |
| *** jpena is now known as jpena|off | 16:10 | |
| mordred | zbr|flu: have you considered not working while you have flu? I think it's good to get rest and drink fluids and not battle python software versions :) | 16:11 |
| * mordred says this knowing he also tends to still work while sick | 16:11 | |
| zbr|flu | ... no comment | 16:11 |
| mordred | clarkb, corvus: the openstacksdk install on the executors that is used in the upload to swift role -that's just the sdk that's installed on the executors itself, right? | 16:12 |
| clarkb | mordred: I think it is installed to the ansible venvs | 16:13 |
| mordred | ah yes | 16:13 |
| *** gfidente has quit IRC | 16:13 | |
| corvus | yeah that | 16:13 |
| clarkb | if you run `grep c15b22fc68d944c38e29e63a23b1452e /var/log/zuul/executor-debug.log` on ze05 you'll see a post failure log for a zuul swift upload job | 16:13 |
| *** bhavikdbavishi has quit IRC | 16:13 | |
| mordred | clarkb: what if I don't want to see that :) | 16:14 |
| clarkb | module_stderr: "WARNING:keystoneauth.discover:Failed to contact the endpoint | 16:14 |
| clarkb | seems to be the tl;dr but there is a traceback that needs formatting too | 16:14 |
| mordred | that warning shoudl go away with sdk v0.34 (but is harmless) | 16:14 |
| *** lucasagomes has quit IRC | 16:15 | |
| clarkb | keystoneauth1.exceptions.http.NotFound: Not Found (HTTP 404)\n" is the end of the traceback | 16:15 |
| mordred | hrm - yeah - that bothers me a bit more | 16:15 |
| clarkb | doesn't seem to log which region that was against | 16:15 |
| *** dave-mccowan has joined #openstack-infra | 16:15 | |
| clarkb | corvus: ^ that might be a good improvement to log which random selectio nwas made (or if we already do that I need to look more closely) | 16:15 |
| corvus | ord i assume because of the helpful keystone warning | 16:15 |
| clarkb | ah ok ya it is in the traceback warning area | 16:16 |
| corvus | that looks like the kind of error we can't do anything about? except maybe retry? | 16:16 |
| clarkb | ya retrying may be the ticket there | 16:17 |
| clarkb | (is it possible that is us getting rate limited?) | 16:17 |
| mordred | corvus: the only place I see us calling HEAD is https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-swift/library/zuul_swift_upload.py#L512-L513 | 16:17 |
| mordred | clarkb: rate limiting looks different | 16:17 |
| clarkb | mordred: k | 16:17 |
| mordred | corvus: we have all the containers in the cdn thing in ord yes? | 16:18 |
| corvus | mordred: it makes the containers as it ges | 16:18 |
| corvus | goes | 16:18 |
| mordred | then great! | 16:18 |
| mordred | oh - maybe eventual consistency | 16:19 |
| clarkb | oh yes! | 16:19 |
| clarkb | I remember swift talking about this in their meeting the other day | 16:19 |
| mordred | so in fact retrying might be a very good idea | 16:19 |
| clarkb | swift currently returns 404 | 16:19 |
| clarkb | but want to chagne to 50something | 16:19 |
| clarkb | so a retry is in fact the right choice there | 16:19 |
| clarkb | timburke: tdasilva ^ can likely tell us all the details on that | 16:19 |
| corvus | so is the theory that we just created the container, enabled cdn on it, then when we performed a head to get the cns url, we got a 404 because it hasn't been set up yet, but if we did it again later, we'd get a real value? | 16:21 |
| clarkb | corvus: yes | 16:22 |
| mordred | corvus: that's the theory | 16:22 |
| clarkb | let me see if I can find the meeting logs | 16:22 |
| corvus | if so, we'll get this error 300 times before all the containers are set up | 16:22 |
| corvus | should we revert now? | 16:22 |
| clarkb | I think http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-07-31-21.00.log.html#l-13 is related | 16:23 |
| clarkb | corvus: can we precreate them? | 16:23 |
| corvus | clarkb: i'm not interested in doing that | 16:23 |
| mordred | corvus: reverting while we add the retry seems liek the least stressful thing | 16:24 |
| clarkb | ok in that case maybe a revert then reenable with retries is the best option | 16:24 |
| corvus | (if we just pre-create them now, the next time this happens, someone will be sad) | 16:24 |
| corvus | (so we should fix it) | 16:24 |
| mordred | ++ | 16:24 |
| mordred | ++ | 16:25 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Revert "Store logs in swift" https://review.opendev.org/676454 | 16:25 |
| mordred | gah | 16:25 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for remoteDispatchDomainMigratePerform3Params lock bug 1840159 https://review.opendev.org/676455 | 16:26 |
| openstack | bug 1840159 in OpenStack Compute (nova) "nova-grenade-live-migration intermittently fails with "Error monitoring migration: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params)"" [Undecided,New] https://launchpad.net/bugs/1840159 | 16:26 |
| mordred | clarkb, corvus: gerrit has stopped loading for me (yay island) - please go ahead and +A the revert | 16:27 |
| mordred | nm - there it is | 16:27 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 16:28 |
| *** ricolin has quit IRC | 16:29 | |
| *** dougsz has joined #openstack-infra | 16:32 | |
| dougsz | https://review.opendev.org/#/c/675574/ | 16:32 |
| dougsz | Is this the place to find a kind soul who can merge a change to enable a Review-Priority flag? ^ | 16:33 |
| clarkb | dougsz: yes | 16:33 |
| dougsz | thanks clarkb | 16:34 |
| *** ijw_ has quit IRC | 16:34 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create zookeeper operator and zuul CR to k8s test https://review.opendev.org/676458 | 16:35 |
| *** dklyle has quit IRC | 16:36 | |
| openstackgerrit | Merged zuul/zuul master: Make tenant and pipeline optional in zuul-changes https://review.opendev.org/674034 | 16:36 |
| *** dklyle has joined #openstack-infra | 16:37 | |
| *** ociuhandu_ has joined #openstack-infra | 16:41 | |
| *** fungi has quit IRC | 16:42 | |
| tdasilva | mordred, clarkb: yeah, a retry with exponential backoff (especially if getting a 5xx error) sounds like the right choice | 16:42 |
| *** fungi has joined #openstack-infra | 16:43 | |
| *** ociuhandu has quit IRC | 16:44 | |
| *** markvoelker has quit IRC | 16:44 | |
| corvus | i guess we don't have pipeline supercedence set up in the opendev tenant? | 16:44 |
| *** ociuhandu_ has quit IRC | 16:45 | |
| mordred | corvus: we should do that | 16:47 |
| mordred | corvus: https://review.opendev.org/676462 Make gate superceede check pipeline | 16:50 |
| *** e0ne has quit IRC | 16:51 | |
| *** markvoelker has joined #openstack-infra | 16:51 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 16:52 |
| *** diablo_rojo has quit IRC | 16:52 | |
| mnaser | dougsz: +w :) | 16:53 |
| dougsz | much appreciated mnaser! | 16:54 |
| *** kopecmartin is now known as kopecmartin|off | 16:54 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: Enable tox-molecule on ensure-tox https://review.opendev.org/672760 | 16:55 |
| *** ociuhandu has joined #openstack-infra | 16:56 | |
| *** psachin has quit IRC | 16:57 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 16:58 |
| *** tdasilva has quit IRC | 16:58 | |
| *** derekh has quit IRC | 17:00 | |
| *** ociuhandu has quit IRC | 17:00 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:01 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:02 |
| openstackgerrit | Merged openstack/project-config master: Enable Review Priority flag for Monasca https://review.opendev.org/675574 | 17:02 |
| *** goldyfruit has quit IRC | 17:03 | |
| *** diablo_rojo has joined #openstack-infra | 17:05 | |
| *** ijw has joined #openstack-infra | 17:05 | |
| *** goldyfruit has joined #openstack-infra | 17:06 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:08 |
| *** diablo_rojo has quit IRC | 17:10 | |
| *** diablo_rojo has joined #openstack-infra | 17:12 | |
| *** jcoufal has joined #openstack-infra | 17:12 | |
| *** spsurya has quit IRC | 17:14 | |
| *** weifan has joined #openstack-infra | 17:15 | |
| *** weifan has quit IRC | 17:20 | |
| *** tdasilva has joined #openstack-infra | 17:23 | |
| *** diablo_rojo has quit IRC | 17:24 | |
| *** diablo_rojo has joined #openstack-infra | 17:24 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add release note for yarn dependencies missing https://review.opendev.org/676468 | 17:25 |
| *** bobh has joined #openstack-infra | 17:28 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:32 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 17:32 |
| *** weifan has joined #openstack-infra | 17:32 | |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: projects: add openstack/ansible-role-uwsgi to zuul https://review.opendev.org/676422 | 17:37 |
| *** priteau has quit IRC | 17:37 | |
| openstackgerrit | Merged opendev/base-jobs master: Revert "Store logs in swift" https://review.opendev.org/676454 | 17:38 |
| *** adriancz has quit IRC | 17:40 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Sync fetch-sphinx-tarball with fetch-sphinx-output https://review.opendev.org/676472 | 17:41 |
| AJaeger | corvus: to make it easy to switch from fetch-sphinx-output to fetch-sphinx-tarball, let's add the new variable as well ^ | 17:41 |
| *** dougsz has quit IRC | 17:41 | |
| corvus | AJaeger: well, i tried to make fetch-sphinx-tarball simpler | 17:42 |
| corvus | i think the variables in fetch-output are incomplehensible | 17:42 |
| corvus | incomprehensible | 17:42 |
| corvus | do we really *need* that? | 17:42 |
| corvus | AJaeger: oh, i think i misunderstood | 17:43 |
| AJaeger | corvus: we need the html for openstack-manuals... | 17:43 |
| corvus | AJaeger: i read "sync" and assumed you brought over all the other variables | 17:43 |
| corvus | AJaeger: but you're just adding the one new one for html | 17:43 |
| AJaeger | corvus: yes, partial sync ;) | 17:43 |
| AJaeger | let me update | 17:43 |
| corvus | that sounds better :) | 17:43 |
| *** dpawlik has joined #openstack-infra | 17:43 | |
| corvus | yeah, i think this makes sense | 17:43 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Partial sync fetch-sphinx-tarball with fetch-sphinx-output https://review.opendev.org/676472 | 17:44 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:44 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 17:44 |
| AJaeger | should we merge your change https://review.opendev.org/676437 ? Then let's ask for additional reviews... | 17:44 |
| corvus | mordred: ^ | 17:48 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 17:49 |
| mordred | corvus, AJaeger: +A | 17:49 |
| corvus | mordred: and https://review.opendev.org/676472 too | 17:49 |
| AJaeger | thanks | 17:50 |
| mordred | ooh - that's great | 17:50 |
| *** diablo_rojo has quit IRC | 17:51 | |
| corvus | mordred: (note it doesn't obviate the need for the deprecation notice, since there's still several other variables that it doesn't have) | 17:51 |
| * AJaeger checks whether we can use opendev-tox-docs for openstack-manuals ;) | 17:51 | |
| corvus | so this basically makes it easier to use in the jobs like AJaeger was showing us earlier (the ones not inherited from tox-docs) | 17:52 |
| corvus | mordred: do you agree with all that? i'm hovering over the approve button for your -announce email and just want to double check before i hit it :) | 17:52 |
| AJaeger | corvus: yes, I need it for https://review.opendev.org/676473 | 17:52 |
| *** tdasilva has quit IRC | 17:53 | |
| mordred | corvus: yeah - I think we should still send the deprecate notice | 17:55 |
| corvus | that revert took a long time to merge, and we didn't get a lot more post failures | 17:56 |
| mordred | corvus: maybe our working theory as to cause was bong and it was just a random 404 | 17:57 |
| corvus | so i suspect that error is more rare that we feared. but still, i'll add a retry in after i get back from lunch before we switch back again | 17:57 |
| mordred | ++ | 17:57 |
| corvus | mordred: could be -- or the theory could be sound it's just that normally the system is fast enough? | 17:57 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Log errors better in case of unknown REST errors https://review.opendev.org/676476 | 17:57 |
| mordred | corvus: yah, maybe that too | 17:57 |
| *** weifan has quit IRC | 17:58 | |
| *** weifan has joined #openstack-infra | 17:58 | |
| mordred | corvus: there's a stab at maybe making the log errors and whatnot better if a similar thing happens in the future - feel free to ignore though - was mostly just thinking out loud | 17:58 |
| corvus | mordred: i like it; i'll incorporate it into my local testing for the retry thing; maybe we can merge both | 17:59 |
| mordred | coolio | 17:59 |
| corvus | i have to afk now, back in a while | 17:59 |
| openstackgerrit | Merged zuul/zuul-jobs master: Return preview artifact in fetch-sphinx-output https://review.opendev.org/676437 | 18:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Partial sync fetch-sphinx-tarball with fetch-sphinx-output https://review.opendev.org/676472 | 18:00 |
| *** weifan has quit IRC | 18:05 | |
| *** weifan has joined #openstack-infra | 18:06 | |
| *** gyee has joined #openstack-infra | 18:06 | |
| *** weifan has quit IRC | 18:07 | |
| *** e0ne has joined #openstack-infra | 18:08 | |
| *** diablo_rojo has joined #openstack-infra | 18:08 | |
| *** e0ne has quit IRC | 18:10 | |
| *** rlandy|rover is now known as rlandy|rover|brb | 18:10 | |
| *** ociuhandu has joined #openstack-infra | 18:11 | |
| *** yamamoto has joined #openstack-infra | 18:11 | |
| *** eharney_ has joined #openstack-infra | 18:15 | |
| *** dpawlik has quit IRC | 18:15 | |
| *** eharney has quit IRC | 18:17 | |
| AJaeger | corvus: fetch-sphinx-tarball needs some more love with the way we use publish-docs in https://review.opendev.org/676473 - more time than I have today... | 18:19 |
| *** diablo_rojo has quit IRC | 18:19 | |
| *** rlandy|rover|brb is now known as rlandy|rover | 18:26 | |
| *** ykarel|away has quit IRC | 18:26 | |
| *** slaweq has joined #openstack-infra | 18:29 | |
| openstackgerrit | David Moreau Simard proposed opendev/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.opendev.org/528739 | 18:29 |
| AJaeger | corvus: let me try switching from publish-docs to publish-docs/html - that will make the fetch-sphinx-tarball and promote jobs much easier... | 18:30 |
| openstackgerrit | Merged zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 18:31 |
| *** ociuhandu has quit IRC | 18:34 | |
| *** mattw4 has quit IRC | 18:41 | |
| *** mattw4 has joined #openstack-infra | 18:44 | |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Allow not writing root-marker file for docs publishing https://review.opendev.org/676488 | 18:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add promote-openstack-manuals jobs https://review.opendev.org/676489 | 18:50 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Allow not writing root-marker file for docs publishing https://review.opendev.org/676488 | 18:51 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add promote-openstack-manuals jobs https://review.opendev.org/676489 | 19:04 |
| AJaeger | corvus: what do you think about these? We might be able to revert the sphinx_output_dir change and make jobs easier this way... | 19:05 |
| AJaeger | ... with some work in openstack-manuals like https://review.opendev.org/676482 | 19:05 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 19:09 |
| *** gagehugo has quit IRC | 19:09 | |
| *** tdasilva has joined #openstack-infra | 19:13 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 19:14 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 19:14 |
| *** beagles_pto has quit IRC | 19:26 | |
| *** kaisers has quit IRC | 19:32 | |
| *** kaisers has joined #openstack-infra | 19:34 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create zookeeper operator and zuul CR to k8s test https://review.opendev.org/676458 | 19:36 |
| *** eharney_ has quit IRC | 19:38 | |
| AJaeger | yeah, https://zuul.opendev.org/t/openstack/build/f2bdbdfbb5fb428b99c5849554fd3005 works - the new openstack-manuals jobs... | 19:42 |
| AJaeger | corvus: ^ | 19:43 |
| corvus | AJaeger, mordred: i'm back | 19:44 |
| AJaeger | corvus: I just tried converting openstack-manuals to promote publishing. If you have time to review jobs today, would be nice - but no urgency. | 19:46 |
| *** goldyfruit has quit IRC | 19:46 | |
| *** goldyfruit has joined #openstack-infra | 19:46 | |
| corvus | AJaeger: when you say we might be able to revert the sphinx_output_dir change -- do you mean https://review.opendev.org/676437 ? | 19:47 |
| AJaeger | Ah, reviewed now ;) I mean we can revert the setting of sphinx_output_dir in that change. We still need the rest the zuul_return one bu tcan get rid of the new variable (and my followup change can be removed completely) | 19:50 |
| corvus | AJaeger: but the new variable is sphinx_build_dir | 19:50 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 19:50 |
| corvus | AJaeger: so i'm a little confused :) | 19:51 |
| *** xenos76 has joined #openstack-infra | 19:51 | |
| AJaeger | corvus: sphinx_output_dir in https://review.opendev.org/#/c/676437/1/roles/fetch-sphinx-output/README.rst | 19:51 |
| corvus | i may need it explained like i've been away for 2 hours | 19:51 |
| corvus | AJaeger: okay, so you're saying there is a way to remove sphinx_output_dir from both roles? | 19:52 |
| AJaeger | corvus: I've been experimenting, so might be confused as well... | 19:52 |
| AJaeger | corvus: yes, with https://review.opendev.org/676482 - I change output from publish-docs to publish-docs/html. | 19:53 |
| AJaeger | Makes everything much easier ;) | 19:53 |
| corvus | AJaeger: oooooh i see | 19:53 |
| AJaeger | corvus: but now I wonder why https://review.opendev.org/676482 does not use a tarball... | 19:54 |
| AJaeger | argh, ignore me - wrong change.... | 19:55 |
| AJaeger | all fine ;) | 19:55 |
| corvus | AJaeger: i agree that makes the jobs changes simpler by standardizing everything on html/ (as was basically the expectation in the tarball job). it's a lot of changes to the manual jobs though, so i didn't want to press it. honestly, i think the outdir_dir change is okay, i'm happy to keep it. if you prefer this, i'm okay with that too, but if someone were to come along later and ask to add | 19:55 |
| corvus | output_dir, i think we'd still probably say yes. :) | 19:55 |
| AJaeger | corvus: the fetch-sphinx-tarball has "html" hardcoded, changing that to publish-docs looked like more work - and all that complexity just for my special jobs is not worth it ;) | 19:56 |
| corvus | AJaeger, mordred: so iiuc, the question is -- do we keep these changes in place: https://review.opendev.org/#/c/676437/1/roles/fetch-sphinx-output/README.rst -- or do we revert that and instead make changes like https://review.opendev.org/676482 | 19:56 |
| AJaeger | corvus: output_dir will not work with sphinx-tarball, so needs more changes | 19:57 |
| corvus | AJaeger: you just changed fetch-sphinx-tarball to not have html hardcoded? | 19:57 |
| corvus | that was this change: https://review.opendev.org/676472 | 19:57 |
| AJaeger | corvus: let me remember what is broken... | 19:58 |
| corvus | AJaeger: the change to add sphinx_output_dir to fetch-sphinx-output plus changes like | 19:58 |
| corvus | grr | 19:58 |
| corvus | AJaeger: the change to add sphinx_output_dir to fetch-sphinx-output plus changes like https://review.opendev.org/676438 should make the manuals jobs work with fetch-sphinx-output | 19:59 |
| corvus | AJaeger: but since you also made https://review.opendev.org/676472 -- that should mean that you can switch to fetch-sphinx-tarball now too, right? | 19:59 |
| AJaeger | yes! | 20:00 |
| AJaeger | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-sphinx-tarball/tasks/main.yaml#L12 is the hardcoded 'html' | 20:01 |
| corvus | oh i didn't see that :) | 20:01 |
| AJaeger | corvus: check the output of that job | 20:01 |
| corvus | AJaeger: i think i was thinking ahead to having that job do smart things with pdfs, etc | 20:01 |
| AJaeger | corvus: neither did I - but when debugging https://review.opendev.org/676483 I found the problem... | 20:01 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Add appending yaml log plugin https://review.opendev.org/623256 | 20:02 |
| corvus | AJaeger: i think i'm starting to understand why https://review.opendev.org/676482 might be the better way now | 20:03 |
| AJaeger | corvus: https://review.opendev.org/#/c/676473/2 showed that problem - | 20:03 |
| corvus | AJaeger: so is the best path forward to revert https://review.opendev.org/676472 and go with https://review.opendev.org/676482 ? | 20:04 |
| AJaeger | corvus: so, please review https://review.opendev.org/676488 and https://review.opendev.org/676489 as well - and we can go with reverting 676472 and merging 676482 | 20:05 |
| AJaeger | those two others are the promote jobs... If we change the build jobs, we either update the publish jobs or add promote ones | 20:05 |
| corvus | AJaeger: why does openstack-manuals need to write the root marker itself? | 20:06 |
| AJaeger | I need to figure out proper order to merge to not create funny directories on docs.o.o - will do that tomorrow with a fresh head | 20:06 |
| AJaeger | corvus, it comes from beeing top-level directory - and publishing branches. | 20:07 |
| AJaeger | Those branches are semi-retired and once we retire them, we could rework... | 20:08 |
| corvus | AJaeger: i think that's safe -- i think the worst that can happen if a job sets that when they shouldn't is that they delete all of their own content | 20:09 |
| AJaeger | ;) | 20:09 |
| corvus | ie, i don't think they can cause any other project's content to be deleted | 20:09 |
| corvus | does that sound right? | 20:09 |
| AJaeger | yes, it does - we still control the publish job and the paths where they publish | 20:10 |
| corvus | (ie, i think if a project set that, and then we published the site index, it might descend down into that project and delete its content because it lacks a root marker) | 20:10 |
| AJaeger | agreed | 20:10 |
| corvus | AJaeger: i +2d that (but maybe we want to add an extra warning or something? :) and i also agree that 676489 looks good. | 20:11 |
| corvus | AJaeger: so how about we go ahead and revert the fetch tarball change so it doesn't get used | 20:12 |
| corvus | AJaeger: and then resume this tomorrow? | 20:12 |
| AJaeger | corvus: sounds good - if you have ideas for wording, go ahead. I'll propose revert... | 20:12 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: DO NO MERGE test gitea http timeouts https://review.opendev.org/676510 | 20:13 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Revert "Partial sync fetch-sphinx-tarball with fetch-sphinx-output" https://review.opendev.org/676511 | 20:13 |
| clarkb | I've not had time to do more proper testing but ^ is something I can do with the time I do have so giving that a go | 20:13 |
| AJaeger | corvus: there's the revert - I sign off for today... | 20:13 |
| AJaeger | Bye | 20:13 |
| corvus | AJaeger: thanks! | 20:13 |
| corvus | mordred: can you +3 https://review.opendev.org/676511 ? | 20:14 |
| corvus | mordred: the really quick summary is that it's probably a bad idea because of things we will want to do in the future, so we should revert it until we work that out tomorrow to make sure no one uses it. | 20:14 |
| mordred | ++ | 20:14 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Allow not writing root-marker file for docs publishing https://review.opendev.org/676488 | 20:14 |
| *** e0ne has joined #openstack-infra | 20:14 | |
| AJaeger | fixed quickly ^ | 20:15 |
| *** ijw has quit IRC | 20:17 | |
| *** diablo_rojo has joined #openstack-infra | 20:20 | |
| *** ijw has joined #openstack-infra | 20:24 | |
| *** ijw has quit IRC | 20:27 | |
| *** ijw has joined #openstack-infra | 20:28 | |
| *** ijw has quit IRC | 20:30 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Partial sync fetch-sphinx-tarball with fetch-sphinx-output" https://review.opendev.org/676511 | 20:30 |
| *** ijw has joined #openstack-infra | 20:30 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Set failed, unreachable, skipped statuses in json plugin https://review.opendev.org/676516 | 20:33 |
| openstackgerrit | Merged zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 20:36 |
| *** diablo_rojo has quit IRC | 20:37 | |
| *** diablo_rojo has joined #openstack-infra | 20:37 | |
| *** ijw has quit IRC | 20:38 | |
| *** ijw has joined #openstack-infra | 20:38 | |
| *** diablo_rojo has quit IRC | 20:41 | |
| *** dpawlik has joined #openstack-infra | 20:45 | |
| *** gagehugo has joined #openstack-infra | 20:46 | |
| *** diablo_rojo has joined #openstack-infra | 20:48 | |
| *** rfolco has quit IRC | 20:51 | |
| *** guoqiao has joined #openstack-infra | 20:52 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Set failed, unreachable, skipped statuses in json plugin https://review.opendev.org/676516 | 20:57 |
| *** kjackal has joined #openstack-infra | 20:58 | |
| *** jtomasek has quit IRC | 21:01 | |
| *** diablo_rojo has quit IRC | 21:04 | |
| *** eharney_ has joined #openstack-infra | 21:05 | |
| *** slaweq has quit IRC | 21:06 | |
| *** diablo_rojo has joined #openstack-infra | 21:09 | |
| *** jcoufal has quit IRC | 21:10 | |
| *** e0ne has quit IRC | 21:11 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for remoteDispatchDomainMigratePerform3Params lock bug 1840159 https://review.opendev.org/676455 | 21:12 |
| openstack | bug 1840159 in OpenStack Compute (nova) "nova-grenade-live-migration intermittently fails with "Error monitoring migration: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params)"" [Undecided,New] https://launchpad.net/bugs/1840159 | 21:12 |
| *** e0ne has joined #openstack-infra | 21:13 | |
| *** e0ne has quit IRC | 21:13 | |
| *** yamamoto has quit IRC | 21:14 | |
| *** jeliu_ has quit IRC | 21:16 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Log errors better in case of unknown REST errors https://review.opendev.org/676476 | 21:22 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry more operations https://review.opendev.org/676518 | 21:22 |
| *** markvoelker has quit IRC | 21:26 | |
| pabelanger | /q dmsimard | 21:27 |
| pabelanger | err | 21:27 |
| *** xenos76 has quit IRC | 21:27 | |
| *** dpawlik has quit IRC | 21:35 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry more operations https://review.opendev.org/676518 | 21:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry more operations https://review.opendev.org/676518 | 21:42 |
| *** dpawlik has joined #openstack-infra | 21:52 | |
| *** mriedem has quit IRC | 21:53 | |
| *** rfolco has joined #openstack-infra | 21:56 | |
| *** dpawlik has quit IRC | 21:56 | |
| *** mattw4 has quit IRC | 21:57 | |
| *** mattw4 has joined #openstack-infra | 21:58 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Revert "Revert "Store logs in swift"" https://review.opendev.org/676523 | 21:59 |
| *** bobh has quit IRC | 22:02 | |
| clarkb | dmsimard: comment on your mirror change (points out why ci failed) | 22:02 |
| *** diablo_rojo has quit IRC | 22:05 | |
| *** diablo_rojo has joined #openstack-infra | 22:05 | |
| *** ekultails has quit IRC | 22:06 | |
| corvus | i think we're all set to merge the retry changes to the swift role in zuul jobs tomorrow and then switch back to swift again | 22:07 |
| corvus | probably too late to merge those today | 22:07 |
| donnyd | speaking of swifty things, I think I am finally good to go on the FN side of it | 22:07 |
| donnyd | just waiting on the nodes to finish a reboot cycle to make sure it all comes back up like it should | 22:08 |
| donnyd | looks like its all working to me, if someone else wants to give testing a go | 22:16 |
| corvus | donnyd: i can run a manual test real quick | 22:17 |
| donnyd | and there should be around 15Tb of space, so hopefully that falls into the spec | 22:18 |
| corvus | donnyd: when i try to list the containers as the openstackzuul user, i get: openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/, ForbiddenAccess was denied to this resource. | 22:21 |
| donnyd | oh i bet that user doesn't have permission | 22:21 |
| donnyd | one sec | 22:21 |
| donnyd | try again | 22:22 |
| corvus | better! | 22:23 |
| donnyd | nice :) :) :) | 22:23 |
| *** markvoelker has joined #openstack-infra | 22:23 | |
| donnyd | hopefully it doesn't blow up on me this time.. I went simple and just used an uncomplicated deployment... Won't perform as good as it could, but I am hopeful for good enough | 22:24 |
| corvus | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/jeblairtest1/ looks good | 22:24 |
| donnyd | hey look at that website, super slick and just worked oob | 22:25 |
| corvus | the Access-Control-Allow-Origin: * | 22:25 |
| corvus | header looks ok | 22:25 |
| corvus | i think that means it has passed the extensive battery of pre-production checks we've come up with :) | 22:26 |
| donnyd | LOL | 22:26 |
| corvus | i will respin the revert revert change to include fn | 22:27 |
| corvus | (also, i just deleted the test container) | 22:27 |
| donnyd | Awesome corvus :) | 22:27 |
| donnyd | so is this already up and running elsewhere? | 22:27 |
| donnyd | and I am curious if logs from FN will only be from FN jobs, or a replica of all jobs | 22:28 |
| corvus | donnyd: it was this morning, then we got a transient error from rax, so we pulled it to add more retry logic; expect it to go back into prod tomorrow | 22:28 |
| corvus | donnyd: the log distribution will be random. we'll only have one copy of each build's logs, and the provider will be random | 22:28 |
| donnyd | it would be really awesome to dynamically link and retain at least two copies | 22:29 |
| donnyd | however we can keep that for another day maybe | 22:29 |
| corvus | yeah, that's worth looking into | 22:30 |
| corvus | atm, we're just really excited about having enough space to expand our retention past a few weeks :) | 22:30 |
| *** eharney_ has quit IRC | 22:30 | |
| corvus | if we lose a few logs along the way, we'll live | 22:31 |
| donnyd | say FN is down for a day due to something beyond my control, well users wouldn't have access to the logs till service it returned... but it is exciting to have hopefully longer term storage of the log data | 22:31 |
| *** kjackal has quit IRC | 22:32 | |
| donnyd | and it will be fun to work with the log data, maybe we can hook up some system that can give us some trend analysis or something even more fancy | 22:32 |
| donnyd | how long are we currently planning to retain logs? | 22:33 |
| corvus | oh you know what, we had talked about it being less, but i think it's currently set to the default of 6 months | 22:34 |
| corvus | if we do want it to be less, now would be a good time to change that :) | 22:34 |
| donnyd | just an FYI I am doing an inplace upgrade on my hypervisors (again) do to local nvme storgae | 22:34 |
| donnyd | so that is why node counts are down to 45 on FN | 22:34 |
| donnyd | eh, 6 months is enough time to get some use from it and honestly I don't see it taking up more than a few TB's | 22:35 |
| donnyd | maybe 4-6 | 22:35 |
| donnyd | I would honestly be interested in keeping them for a year so we could provide insights over a longer period of time... even if it costs a few more TB of space | 22:36 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add FN to swift log storage https://review.opendev.org/676527 | 22:38 |
| corvus | i did that as a followup commit to make reverting easier if necessary | 22:39 |
| corvus | but i think we can merge it right after its parent | 22:39 |
| *** dchen has joined #openstack-infra | 22:40 | |
| clarkb | I +2'd the parent change but did not approve as I can't watch it currently | 22:42 |
| *** alemgeta has quit IRC | 22:42 | |
| *** markvoelker has quit IRC | 22:48 | |
| *** tkajinam has joined #openstack-infra | 22:50 | |
| *** tyreymer has joined #openstack-infra | 23:05 | |
| *** rlandy|rover has quit IRC | 23:12 | |
| *** yamamoto has joined #openstack-infra | 23:13 | |
| *** diablo_rojo has quit IRC | 23:18 | |
| *** diablo_rojo has joined #openstack-infra | 23:18 | |
| *** markvoelker has joined #openstack-infra | 23:25 | |
| *** ijw has quit IRC | 23:29 | |
| *** markvoelker has quit IRC | 23:30 | |
| *** diablo_rojo has quit IRC | 23:31 | |
| *** sshnaidm is now known as sshnaidm|afk | 23:38 | |
| *** tyreymer has quit IRC | 23:41 | |
| *** sthussey has quit IRC | 23:53 | |
| *** pkopec has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!