| *** jamesmcarthur has joined #openstack-infra | 00:02 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Test swift upload logs without encoding type for gzip files https://review.opendev.org/701283 | 00:04 |
|---|---|---|
| *** eernst has joined #openstack-infra | 00:11 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** eernst has quit IRC | 00:15 | |
| *** slaweq has quit IRC | 00:15 | |
| openstackgerrit | Ghanshyam Mann proposed opendev/irc-meetings master: Update QA office hour time https://review.opendev.org/701251 | 00:29 |
| clarkb | I have restored https://review.opendev.org/#/c/680178/ to start testing the gzip.changes but I dont think that tries to upload any gzipped files? | 00:41 |
| *** ricolin_ has joined #openstack-infra | 00:44 | |
| *** ricolin_ has quit IRC | 00:45 | |
| openstackgerrit | Rico Lin proposed opendev/irc-meetings master: Combine auto-scaling and self-healing SIG meeting https://review.opendev.org/701137 | 00:50 |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| *** igordc has joined #openstack-infra | 01:49 | |
| *** stewie925 has quit IRC | 01:51 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** zbr_ has joined #openstack-infra | 02:04 | |
| *** zbr has quit IRC | 02:07 | |
| *** zbr has joined #openstack-infra | 02:09 | |
| *** zbr_ has quit IRC | 02:11 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** Goneri has quit IRC | 02:19 | |
| *** dSrinivas has joined #openstack-infra | 02:26 | |
| *** goldyfruit_ has joined #openstack-infra | 02:35 | |
| *** goldyfruit_ has quit IRC | 02:36 | |
| *** goldyfruit_ has joined #openstack-infra | 02:36 | |
| *** igordc has quit IRC | 02:43 | |
| *** michael-beaver has quit IRC | 03:05 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** rcernin_ has joined #openstack-infra | 03:13 | |
| *** rcernin has quit IRC | 03:13 | |
| *** slaweq has quit IRC | 03:16 | |
| *** hongbin has joined #openstack-infra | 03:34 | |
| *** chandankumar has joined #openstack-infra | 03:55 | |
| *** rcernin_ has quit IRC | 03:56 | |
| *** rcernin has joined #openstack-infra | 04:00 | |
| *** udesale has joined #openstack-infra | 04:01 | |
| *** ykarel has joined #openstack-infra | 04:04 | |
| *** ykarel has quit IRC | 04:14 | |
| *** goldyfruit_ has quit IRC | 04:24 | |
| *** goldyfruit_ has joined #openstack-infra | 04:24 | |
| *** hongbin has quit IRC | 04:24 | |
| *** spsurya has joined #openstack-infra | 04:32 | |
| *** chandankumar has quit IRC | 04:34 | |
| *** ykarel has joined #openstack-infra | 04:34 | |
| *** sshnaidm|afk has quit IRC | 04:38 | |
| *** goldyfruit_ has quit IRC | 04:48 | |
| *** tkajinam has quit IRC | 04:52 | |
| *** eernst has joined #openstack-infra | 04:58 | |
| *** gyee has quit IRC | 05:00 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** chandankumar has joined #openstack-infra | 05:13 | |
| *** goldyfruit_ has joined #openstack-infra | 05:13 | |
| *** slaweq has quit IRC | 05:16 | |
| *** tkajinam has joined #openstack-infra | 05:23 | |
| *** surpatil has joined #openstack-infra | 05:25 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** sshnaidm has joined #openstack-infra | 05:30 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:33 | |
| *** eernst has quit IRC | 05:35 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** ykarel_ has joined #openstack-infra | 05:40 | |
| *** ykarel has quit IRC | 05:42 | |
| *** chandankumar has quit IRC | 05:52 | |
| *** sshnaidm has quit IRC | 05:54 | |
| *** tkajinam has quit IRC | 05:58 | |
| *** pcaruana has joined #openstack-infra | 06:00 | |
| *** chandankumar has joined #openstack-infra | 06:02 | |
| *** dave-mccowan has quit IRC | 06:07 | |
| *** pcaruana has quit IRC | 06:14 | |
| *** ykarel__ has joined #openstack-infra | 06:29 | |
| *** ykarel_ has quit IRC | 06:32 | |
| *** ykarel__ is now known as ykarel | 06:33 | |
| *** tkajinam has joined #openstack-infra | 06:41 | |
| *** tkajinam_ has joined #openstack-infra | 06:52 | |
| *** rcernin has quit IRC | 06:54 | |
| *** tkajinam has quit IRC | 06:55 | |
| *** AJaeger has quit IRC | 06:55 | |
| *** dpawlik has joined #openstack-infra | 06:56 | |
| *** ccamacho has joined #openstack-infra | 07:01 | |
| *** lmiccini has joined #openstack-infra | 07:01 | |
| *** AJaeger has joined #openstack-infra | 07:02 | |
| AJaeger | amotoki, ianychoi : *some* translation jobs are failing, some work - for failures, see http://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-update&job_name=upstream-translation-update&result=FAILURE | 07:06 |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** pgaxatte has joined #openstack-infra | 07:16 | |
| *** slaweq has quit IRC | 07:17 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 07:18 |
| AJaeger | config-core, please review https://review.opendev.org/700913, https://review.opendev.org/701244 , https://review.opendev.org/700909 | 07:19 |
| *** SurajPatil has joined #openstack-infra | 07:24 | |
| *** surpatil has quit IRC | 07:27 | |
| ianychoi | amotoki, not too much sure but when I try to search with an error message from the translation job errors: "ImportError: cannot import name 'TOKEN_TEXT'", might that it is related with Python 2 issues in Horizon? | 07:28 |
| *** piotrowskim has joined #openstack-infra | 07:33 | |
| *** ykarel is now known as ykarel|lunch | 07:38 | |
| *** roman_g has joined #openstack-infra | 07:45 | |
| *** roman_g has quit IRC | 07:45 | |
| *** roman_g has joined #openstack-infra | 07:46 | |
| *** kjackal has joined #openstack-infra | 07:48 | |
| *** ralonsoh has joined #openstack-infra | 07:49 | |
| AJaeger | ianychoi: we switched translation jobs yesterday to python3 | 08:01 |
| hrw | AJaeger: can you take a look at https://review.opendev.org/701227 patch? adds mirroring of arm64 packages from ubuntu cloud archive | 08:02 |
| *** slaweq has joined #openstack-infra | 08:02 | |
| *** iurygregory has joined #openstack-infra | 08:02 | |
| *** SurajPatil has quit IRC | 08:08 | |
| AJaeger | hrw: I'm not a core for system-config | 08:10 |
| hrw | AJaeger: ok | 08:10 |
| *** udesale has quit IRC | 08:16 | |
| *** lplearn21 has joined #openstack-infra | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:17 | |
| *** SurajPatil has joined #openstack-infra | 08:18 | |
| *** surpatil has joined #openstack-infra | 08:20 | |
| *** SurajPatil has quit IRC | 08:23 | |
| *** tosky has joined #openstack-infra | 08:23 | |
| *** Buggys has left #openstack-infra | 08:24 | |
| *** arxcruz|off is now known as arxcruz | 08:24 | |
| *** SurajPatil has joined #openstack-infra | 08:25 | |
| *** surpatil has quit IRC | 08:25 | |
| *** dchen has quit IRC | 08:25 | |
| *** pcaruana has joined #openstack-infra | 08:29 | |
| *** FlorianFa has quit IRC | 08:30 | |
| *** SurajPatil has quit IRC | 08:31 | |
| *** surpatil has joined #openstack-infra | 08:31 | |
| *** SurajPatil has joined #openstack-infra | 08:32 | |
| *** sshnaidm has joined #openstack-infra | 08:32 | |
| *** aedc_ has quit IRC | 08:34 | |
| *** ykarel|lunch is now known as ykarel | 08:35 | |
| *** SurajPatil has quit IRC | 08:35 | |
| *** surpatil has quit IRC | 08:35 | |
| *** rpittau|afk is now known as rpittau | 08:36 | |
| *** surpatil has joined #openstack-infra | 08:36 | |
| dSrinivas | AJaeger, How can I run the Jenkins Jobs on Xenial as of now and later i will update the image to bionic. Any workaround please suggest me. Thank you. | 08:38 |
| amotoki | ianychoi: as of now, translation extraction in the master branch is broken. it is caused by that django-babel is not compatible with recent django version as django-babel project is unmaintained for long. we are in the way to switch it to enmerkar (a fork of django-babel). | 08:39 |
| *** SurajPatil has joined #openstack-infra | 08:40 | |
| amotoki | ianychoi: as AJaeger mentioned, we switched the translation jobs to py3 yesterday. TOKEN_EXIT import error still exists. The job itself succeeds but it actually does not work in the master branch. | 08:41 |
| *** surpatil has quit IRC | 08:42 | |
| amotoki | ah, TOKEN_EXIT import error now leads to failure of the jobs. | 08:43 |
| amotoki | it will be fixed by https://review.opendev.org/#/c/701124/ | 08:44 |
| *** xek has joined #openstack-infra | 08:44 | |
| *** priteau has joined #openstack-infra | 08:47 | |
| *** surpatil has joined #openstack-infra | 08:49 | |
| *** chandankumar has quit IRC | 08:51 | |
| *** SurajPatil has quit IRC | 08:52 | |
| *** tridde has quit IRC | 08:53 | |
| *** chandankumar has joined #openstack-infra | 08:53 | |
| AJaeger | amotoki: is that the right link? | 08:54 |
| AJaeger | amotoki: Note that it has "Cannot merge" -> needs rebase | 08:54 |
| *** trident has joined #openstack-infra | 08:55 | |
| *** tkajinam_ has quit IRC | 08:58 | |
| *** sshnaidm has quit IRC | 09:02 | |
| *** slaweq_ has joined #openstack-infra | 09:05 | |
| *** dmellado has quit IRC | 09:07 | |
| *** udesale has joined #openstack-infra | 09:07 | |
| *** slaweq has quit IRC | 09:07 | |
| *** dmellado has joined #openstack-infra | 09:08 | |
| *** lucasagomes has joined #openstack-infra | 09:10 | |
| openstackgerrit | Ilya Etingof proposed openstack/project-config master: Add `sushy-cli` project https://review.opendev.org/700909 | 09:10 |
| *** gfidente has joined #openstack-infra | 09:34 | |
| *** derekh has joined #openstack-infra | 09:36 | |
| *** dtantsur|afk is now known as dtantsur | 09:40 | |
| *** sshnaidm has joined #openstack-infra | 09:42 | |
| *** aedc has joined #openstack-infra | 09:54 | |
| *** ociuhandu has joined #openstack-infra | 10:00 | |
| *** udesale has quit IRC | 10:08 | |
| *** ociuhandu has quit IRC | 10:08 | |
| *** udesale has joined #openstack-infra | 10:09 | |
| *** rlandy has quit IRC | 10:10 | |
| *** slaweq_ is now known as slaweq | 10:16 | |
| *** udesale has quit IRC | 10:24 | |
| *** pkopec has joined #openstack-infra | 10:26 | |
| *** SurajPatil has joined #openstack-infra | 10:36 | |
| *** priteau has quit IRC | 10:37 | |
| *** ociuhandu has joined #openstack-infra | 10:39 | |
| *** surpatil has quit IRC | 10:39 | |
| *** chandankumar has quit IRC | 10:39 | |
| *** ociuhandu has quit IRC | 10:54 | |
| *** ociuhandu has joined #openstack-infra | 10:55 | |
| *** surpatil has joined #openstack-infra | 10:55 | |
| *** ociuhandu has quit IRC | 10:56 | |
| *** ociuhandu has joined #openstack-infra | 10:56 | |
| *** udesale has joined #openstack-infra | 10:56 | |
| *** SurajPatil has quit IRC | 10:58 | |
| *** mcepl has quit IRC | 11:00 | |
| *** ociuhandu has quit IRC | 11:02 | |
| *** mcepl has joined #openstack-infra | 11:02 | |
| *** ykarel is now known as ykarel|afk | 11:03 | |
| *** ociuhandu has joined #openstack-infra | 11:05 | |
| *** ociuhandu has quit IRC | 11:06 | |
| *** ociuhandu has joined #openstack-infra | 11:06 | |
| *** ykarel|afk has quit IRC | 11:09 | |
| *** ociuhandu has quit IRC | 11:11 | |
| *** udesale has quit IRC | 11:13 | |
| *** rpittau is now known as rpittau|bbl | 11:16 | |
| openstackgerrit | Merged opendev/irc-meetings master: Combine auto-scaling and self-healing SIG meeting https://review.opendev.org/701137 | 11:23 |
| openstackgerrit | Merged opendev/irc-meetings master: Update QA office hour time https://review.opendev.org/701251 | 11:29 |
| *** sshnaidm has quit IRC | 11:32 | |
| *** sshnaidm has joined #openstack-infra | 11:33 | |
| *** Lucas_Gray has joined #openstack-infra | 11:40 | |
| *** ykarel|afk has joined #openstack-infra | 11:43 | |
| *** SurajPatil has joined #openstack-infra | 11:50 | |
| *** surpatil has quit IRC | 11:52 | |
| *** jangutter has joined #openstack-infra | 11:53 | |
| *** aedc has quit IRC | 12:03 | |
| *** surpatil has joined #openstack-infra | 12:09 | |
| *** zbr has quit IRC | 12:11 | |
| *** zbr_ has joined #openstack-infra | 12:11 | |
| *** SurajPatil has quit IRC | 12:12 | |
| *** surpatil has quit IRC | 12:14 | |
| *** surpatil has joined #openstack-infra | 12:15 | |
| *** lpetrut has joined #openstack-infra | 12:33 | |
| *** rfolco has joined #openstack-infra | 12:45 | |
| *** rlandy has joined #openstack-infra | 12:57 | |
| *** dpawlik has quit IRC | 12:59 | |
| *** szaher has joined #openstack-infra | 13:00 | |
| *** dpawlik has joined #openstack-infra | 13:03 | |
| *** udesale has joined #openstack-infra | 13:04 | |
| *** rh-jelabarre has joined #openstack-infra | 13:05 | |
| *** goldyfruit_ has quit IRC | 13:20 | |
| *** aaronsheffield has joined #openstack-infra | 13:22 | |
| *** rpittau|bbl is now known as rpittau | 13:26 | |
| *** dave-mccowan has joined #openstack-infra | 13:28 | |
| *** Lucas_Gray has quit IRC | 13:30 | |
| *** sshnaidm is now known as sshnaidm|mtg | 13:33 | |
| *** ykarel|afk is now known as ykarel | 13:41 | |
| *** Goneri has joined #openstack-infra | 13:42 | |
| *** Lucas_Gray has joined #openstack-infra | 13:42 | |
| *** smarcet has joined #openstack-infra | 13:43 | |
| *** Lucas_Gray has quit IRC | 13:46 | |
| *** priteau has joined #openstack-infra | 13:46 | |
| *** diablo_rojo has joined #openstack-infra | 13:47 | |
| *** liuyulong has joined #openstack-infra | 13:47 | |
| *** Lucas_Gray has joined #openstack-infra | 13:49 | |
| *** ociuhandu has joined #openstack-infra | 13:54 | |
| *** Lucas_Gray has quit IRC | 13:55 | |
| *** surpatil has quit IRC | 13:56 | |
| *** Lucas_Gray has joined #openstack-infra | 13:59 | |
| *** ociuhandu has quit IRC | 14:01 | |
| *** ociuhandu has joined #openstack-infra | 14:11 | |
| *** dpawlik has quit IRC | 14:13 | |
| *** pcaruana has quit IRC | 14:20 | |
| *** eharney has joined #openstack-infra | 14:21 | |
| *** ociuhandu has quit IRC | 14:22 | |
| *** dSrinivas has quit IRC | 14:31 | |
| *** pcaruana has joined #openstack-infra | 14:32 | |
| *** ociuhandu has joined #openstack-infra | 14:33 | |
| *** lbragstad_ is now known as lbragstad | 14:34 | |
| *** pkopec has quit IRC | 14:34 | |
| *** pkopec has joined #openstack-infra | 14:35 | |
| *** lbragstad has quit IRC | 14:35 | |
| *** Lucas_Gray has quit IRC | 14:36 | |
| *** lbragstad has joined #openstack-infra | 14:41 | |
| *** ociuhandu has quit IRC | 14:42 | |
| AJaeger | mnaser: will vexxhost swift be enabled at some time again or do you want to abandon https://review.opendev.org/#/c/678440/ ? | 14:48 |
| AJaeger | config-core, please review https://review.opendev.org/700913, https://review.opendev.org/701244 , and https://review.opendev.org/700909 | 14:49 |
| *** zbr_ is now known as zbr|rover | 14:50 | |
| *** ykarel is now known as ykarel|afk | 14:52 | |
| *** Lucas_Gray has joined #openstack-infra | 14:54 | |
| *** sshnaidm|mtg is now known as sshnaidm|afk | 14:55 | |
| *** jaosorior has joined #openstack-infra | 15:01 | |
| *** jaosorior has quit IRC | 15:01 | |
| *** lplearn21 has quit IRC | 15:01 | |
| dtantsur | mordred or anyone: hey, do you know if Glean works with RHEL/CentOS 8? | 15:02 |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 15:05 |
| *** pgaxatte has quit IRC | 15:08 | |
| *** ociuhandu has joined #openstack-infra | 15:08 | |
| *** ramishra has quit IRC | 15:16 | |
| *** ramishra has joined #openstack-infra | 15:17 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Don't preserve logs uid and guid https://review.opendev.org/701381 | 15:20 |
| *** smarcet has quit IRC | 15:22 | |
| *** ociuhandu has quit IRC | 15:22 | |
| *** michael-beaver has joined #openstack-infra | 15:28 | |
| *** eernst has joined #openstack-infra | 15:30 | |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 15:31 |
| *** lpetrut has quit IRC | 15:31 | |
| *** eernst has quit IRC | 15:31 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add cri-o support to use-buildset registry https://review.opendev.org/701237 | 15:36 |
| openstackgerrit | Merged opendev/system-config master: reprepro: sync arm64 packages from Ubuntu cloud archive https://review.opendev.org/701227 | 15:39 |
| fungi | dtantsur: it should. we're using it for our centos-8 images in nodepool | 15:42 |
| dtantsur | sweeet | 15:42 |
| dtantsur | we may have a use case for it in ironic | 15:42 |
| fungi | i think ianw_pto did some not-insignificant work to solve the centos switch to... networkmanager? | 15:43 |
| dtantsur | yes, I think it uses NM now. | 15:43 |
| fungi | was also necessary for more recent fedora versions, if memory serves | 15:43 |
| dtantsur | cool, thanks! | 15:44 |
| *** udesale has quit IRC | 15:48 | |
| *** ykarel|afk is now known as ykarel|away | 15:49 | |
| *** Lucas_Gray has quit IRC | 15:54 | |
| hrw | thanks for merging 701227 | 15:56 |
| hrw | now I just need to wait for next mirror sync and can continue with ubuntu/aarch64 ci job | 15:56 |
| *** ykarel|away has quit IRC | 16:01 | |
| *** eharney has quit IRC | 16:02 | |
| *** ociuhandu has joined #openstack-infra | 16:05 | |
| *** mcepl has left #openstack-infra | 16:13 | |
| *** ociuhandu has quit IRC | 16:13 | |
| *** ociuhandu has joined #openstack-infra | 16:15 | |
| *** gyee has joined #openstack-infra | 16:16 | |
| *** rfolco is now known as rfolcOUT | 16:16 | |
| clarkb | infra-root https://zuul.opendev.org/t/zuul/build/b0b72b903255497a84c7d024be5e0adf/logs shows we didn't set content-encoding on the gz files so that bit worked, but the site variable to not gzip subunit stuff seems to not be working | 16:17 |
| clarkb | I'll track that down after my morning meeting | 16:17 |
| *** electrofelix has joined #openstack-infra | 16:23 | |
| *** lmiccini has quit IRC | 16:25 | |
| *** ociuhandu has quit IRC | 16:28 | |
| *** KeithMnemonic has quit IRC | 16:30 | |
| *** eharney has joined #openstack-infra | 16:31 | |
| *** KeithMnemonic has joined #openstack-infra | 16:31 | |
| *** ociuhandu has joined #openstack-infra | 16:37 | |
| *** ociuhandu has quit IRC | 16:46 | |
| *** diablo_rojo has quit IRC | 16:52 | |
| *** mattw4 has joined #openstack-infra | 16:53 | |
| mordred | dtantsur: let us know if it doesn't work in the ironic case - we worked with TheJulia ages ago to make sure the config-drive support for static networking worked for bifrost in addition to for rackspace - but that was ages ago | 16:57 |
| dtantsur | cool! | 16:57 |
| TheJulia | many ages ago | 16:57 |
| clarkb | oh I'm a derp re gzipped things | 16:58 |
| clarkb | stage-output is controlled by that flag not fetch-subunit | 16:58 |
| clarkb | if I wget https://b527bd44c15a22b5d896-b5129c1862a1ee160f1f629c32dfbc83.ssl.cf5.rackcdn.com/680178/3/check/tox-py35/32f520b/testr_results.html.gz I get back gzipped data so that bit is working | 16:58 |
| clarkb | corvus: fungi: ^ I think that means we can proceed with merging https://review.opendev.org/#/c/701284/ | 16:59 |
| fungi | yeah, i concur. looking back over it one last time now | 17:00 |
| *** lucasagomes has quit IRC | 17:00 | |
| clarkb | then send an email (to zuul-announce or discuss?) about https://review.opendev.org/#/c/701282/ and merge it? | 17:00 |
| *** hwoarang has quit IRC | 17:01 | |
| *** hwoarang has joined #openstack-infra | 17:04 | |
| *** pcaruana has quit IRC | 17:05 | |
| *** hwoarang has quit IRC | 17:09 | |
| *** rpittau is now known as rpittau|afk | 17:09 | |
| *** hwoarang has joined #openstack-infra | 17:10 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-gate master: Re-enable the n-cauth required service for stable/queens|pike|ocata https://review.opendev.org/701404 | 17:12 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Add basic Helm jobs https://review.opendev.org/700222 | 17:12 |
| clarkb | corvus: fungi https://etherpad.openstack.org/p/wwgBLSq1Rq something like that email maybe? | 17:14 |
| *** ociuhandu has joined #openstack-infra | 17:15 | |
| *** hwoarang has quit IRC | 17:20 | |
| *** hwoarang has joined #openstack-infra | 17:20 | |
| *** gfidente is now known as gfidente|afk | 17:22 | |
| *** tosky has quit IRC | 17:23 | |
| *** ociuhandu has quit IRC | 17:25 | |
| fungi | clarkb: should 701284 extend the documentation at https://zuul-ci.org/docs/zuul-jobs/log-roles.html#rolevar-upload-logs.zuul_log_compress or https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-upload-logs-swift with some of that explanation as well? | 17:26 |
| fungi | also, this discussion is likely more relevant for #zuul | 17:26 |
| *** pcaruana has joined #openstack-infra | 17:28 | |
| *** hamzy has quit IRC | 17:29 | |
| *** ociuhandu has joined #openstack-infra | 17:31 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-infra | 17:33 | |
| *** ociuhandu has quit IRC | 17:37 | |
| *** dtantsur is now known as dtantsur|afk | 17:37 | |
| *** ociuhandu has joined #openstack-infra | 17:39 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Added nodepool to charts https://review.opendev.org/700462 | 17:41 |
| *** aedc has joined #openstack-infra | 17:41 | |
| *** priteau has quit IRC | 17:45 | |
| *** ociuhandu has quit IRC | 17:46 | |
| mordred | fungi, clarkb, corvus, Shrews: anybody want to upgrade gitea? https://review.opendev.org/#/c/699490/ | 17:52 |
| *** kjackal has quit IRC | 17:52 | |
| clarkb | mordred: you've got the +2's but I'm not approving as I have to prep for the meeting, run the meeting, then pick up kids from school so not in a good spot to watch it right now. Feel free to approve if you can watch though | 17:54 |
| mordred | clarkb: cool. I'll +A it after the meeting - I've got a call in 6 minutes | 17:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: Swift upload logs without encoding type for gzip files https://review.opendev.org/701284 | 17:56 |
| *** derekh has quit IRC | 18:00 | |
| *** bdodd has quit IRC | 18:04 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Added nodepool to charts https://review.opendev.org/700462 | 18:13 |
| *** michael-beaver has quit IRC | 18:18 | |
| *** xek has quit IRC | 18:27 | |
| *** rishabhhpe has joined #openstack-infra | 18:33 | |
| *** xek has joined #openstack-infra | 18:37 | |
| *** rishabhhpe has quit IRC | 18:40 | |
| *** trident has quit IRC | 18:48 | |
| *** trident has joined #openstack-infra | 18:49 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Added nodepool to charts https://review.opendev.org/700462 | 18:52 |
| openstackgerrit | Merged zuul/zuul master: Don't set ansible_python_interpreter if in vars https://review.opendev.org/698774 | 18:52 |
| fungi | clarkb: so we can tell #openstack-release we're all clear on compression now? | 19:00 |
| fungi | also, meeting time! | 19:00 |
| clarkb | fungi: yes I think so | 19:01 |
| fungi | thanks! i'll let them know | 19:01 |
| *** hwoarang has quit IRC | 19:10 | |
| *** hwoarang has joined #openstack-infra | 19:11 | |
| *** jtomasek has quit IRC | 19:13 | |
| *** ralonsoh has quit IRC | 19:18 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Remove implemented specs https://review.opendev.org/701435 | 19:19 |
| *** xek has quit IRC | 19:23 | |
| *** xek has joined #openstack-infra | 19:23 | |
| *** bdodd has joined #openstack-infra | 19:24 | |
| *** spsurya has quit IRC | 19:25 | |
| *** hamzy has joined #openstack-infra | 19:26 | |
| timburke | sorry, i've been head-down for a bit -- what's going on with swift and content-encodings? | 19:27 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Added nodepool to charts https://review.opendev.org/700462 | 19:28 |
| clarkb | timburke: if we upload a tarball to some swifts and set content-encoding as gzip when you download that tarball you get it back as a tarfile with no compression if the client doesn't set accept-encoding gzip | 19:28 |
| clarkb | timburke: not sure if that is actually swift doing that or the webservers in front of it | 19:28 |
| timburke | that sounds... a little crazy. what's the client? | 19:28 |
| clarkb | timburke: our solution is to not set content-encoding for files that originate as gzip | 19:28 |
| timburke | i know swiftclient used to have some trouble with that. fixed in https://opendev.org/openstack/python-swiftclient/commit/7d5c85ad1013185bd3aa7f0d384f2e0e68e3b484 | 19:28 |
| clarkb | timburke: wget or curl. basically anything that does simple http requests | 19:29 |
| timburke | crazy! idk then... | 19:29 |
| fungi | it definitely could just be rackspace's stuff causing that | 19:30 |
| fungi | either their swiftish bits or their black box cdn | 19:30 |
| timburke | *nods* i suppose i could see the CDN trying to be "helpful"... | 19:33 |
| *** electrofelix has quit IRC | 19:33 | |
| fungi | we've seen that particular cdn try to be "helpful" in other ways already | 19:33 |
| fungi | around layering deflate compression on top of already compressed files | 19:33 |
| fungi | i'm guessing it's mainly trying to be "helpful" to rackspace's unbillable cdn bandwidth utilization | 19:34 |
| clarkb | fungi: though in this case it is inflating the data transmitted | 19:35 |
| fungi | oh, that's a good point, actually uses more bandwidth (possibly lots more) | 19:36 |
| *** xek has quit IRC | 19:38 | |
| *** xek has joined #openstack-infra | 19:42 | |
| *** hrw has left #openstack-infra | 19:45 | |
| clarkb | infra-root https://review.opendev.org/#/c/701278/1 and its child are passing tests now. After lunch and picking up kids I'll try to get those landed so that we can keep pushing on LE certs for zuul.opendev.org | 19:48 |
| clarkb | that change needs another review though | 19:48 |
| fungi | oh, yep, i had already looked at that one but got distracted by the failing jobs | 19:49 |
| fungi | approved | 19:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire x/dox (step 2) https://review.opendev.org/701446 | 19:53 |
| mordred | infra-root: I have +Ad the gitea upgrade - hopefully nobody will notice | 19:53 |
| fungi | i'll keep an eye (or two) on it | 19:56 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring x/dox (step 4) https://review.opendev.org/701447 | 19:56 |
| *** dpawlik has joined #openstack-infra | 19:56 | |
| AJaeger | config-core, could please review https://review.opendev.org/700913, https://review.opendev.org/701244 , and https://review.opendev.org/700909 ? | 19:57 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-dox-buildimages job https://review.opendev.org/701448 | 19:59 |
| *** dpawlik has quit IRC | 20:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring x/dox (step 4) https://review.opendev.org/701447 | 20:00 |
| AJaeger | clarkb: do you want me to abandon dox changes and self-merge the retirement change? Then, add me to https://review.opendev.org/#/admin/groups/434,members , please. See stack starting at https://review.opendev.org/701446 | 20:05 |
| *** kjackal has joined #openstack-infra | 20:06 | |
| *** slaweq has quit IRC | 20:19 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Add Zuul charts https://review.opendev.org/700460 | 20:25 |
| *** gfidente|afk is now known as gfidente | 20:26 | |
| *** Lucas_Gray has joined #openstack-infra | 20:30 | |
| AJaeger | infra-root, what do you think of https://review.opendev.org/#/c/699697/ to ignore retired repos in jeepyb? | 20:31 |
| fungi | i guess i'm fine with the idea, but if the goal is to eventually replace hound with gitea's upcoming elasticsearch then that'll be a regression (unless they add a feature to allow us to flag specific repos so they won't get indexed there) | 20:35 |
| *** slaweq has joined #openstack-infra | 20:35 | |
| AJaeger | interesting - mordred ^ | 20:36 |
| corvus | i'm fine with the change, and also the regression if/when we move to gitea | 20:37 |
| AJaeger | please review https://review.opendev.org/#/c/699715 as well - it's needed first. | 20:38 |
| mordred | gitea has an "archive" repo that I think we're using when we retire stuff, right? hopefully that would be excluded in search results - or hopefully they'd be amenable to making it that way | 20:39 |
| *** gfidente is now known as gfidente|afk | 20:41 | |
| *** pcaruana has quit IRC | 20:41 | |
| fungi | oh, yeah i could see at least an option to not include archived repos in search results by default as a useful feature | 20:43 |
| AJaeger | fungi, could you review 699715, please? | 20:44 |
| fungi | yeah, approved | 20:45 |
| fungi | thanks! | 20:45 |
| AJaeger | thanks | 20:45 |
| *** eharney has quit IRC | 20:49 | |
| openstackgerrit | Merged opendev/jeepyb master: Fix create_hound_config for python3 https://review.opendev.org/699715 | 20:50 |
| openstackgerrit | Merged opendev/jeepyb master: Don't index retired projects https://review.opendev.org/699697 | 20:50 |
| *** xek has quit IRC | 20:54 | |
| *** slaweq has quit IRC | 20:54 | |
| *** dpawlik has joined #openstack-infra | 20:55 | |
| *** slaweq has joined #openstack-infra | 20:55 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Add Zuul charts https://review.opendev.org/700460 | 20:56 |
| openstackgerrit | Merged opendev/system-config master: Bump gitea version to 1.10.2 https://review.opendev.org/699490 | 20:59 |
| *** Goneri has quit IRC | 21:00 | |
| corvus | clarkb, fungi: it looks like acme ran but LE did not grant us certs for zuul on files02 | 21:07 |
| corvus | i'm unsure how to debug this | 21:08 |
| corvus | or find logs | 21:08 |
| clarkb | corvus: I think logs go in /var/log/acme.sh | 21:08 |
| clarkb | the ansible logs on bridge may also have info | 21:08 |
| corvus | yes there are logs there! | 21:09 |
| *** dpawlik has quit IRC | 21:21 | |
| fungi | however they're not especially helpful logs | 21:22 |
| fungi | the records exist and look correct | 21:23 |
| fungi | the txt values mentioned are present | 21:23 |
| corvus | the logs on files02 says it should be "_acme-challenge.acme.opendev.org." but the txt records are for "acme.opendev.org" | 21:23 |
| clarkb | the txt records arein acme.opendev.org zone? | 21:23 |
| corvus | i'm a little confused | 21:23 |
| fungi | we cname to that, and it's a non-normally-managed domain so that ansible can plug values into it on the fly | 21:24 |
| mordred | opendev.org seems to be serving from 1.10.2 now | 21:25 |
| fungi | however, it should be looking at _acme-challenge.zuul-ci.org which is a cname to acme.opendev.org | 21:25 |
| fungi | and that contains the correct values | 21:25 |
| fungi | *but* | 21:25 |
| fungi | the log says to use _acme-challenge.acme.opendev.org which doesn't work | 21:26 |
| fungi | i wonder why | 21:26 |
| corvus | oh, what if this is it: | 21:26 |
| corvus | ERROR! The requested handler 'letsencrypt updated tarballs-main' was not found in either the main handlers list nor in the listening handlers list | 21:26 |
| corvus | End letsencrypt | 21:26 |
| *** dpawlik has joined #openstack-infra | 21:26 | |
| corvus | maybe ansible stopped early before running the final step | 21:27 |
| clarkb | dont handlers run at the end? but fixing that seems like a good next step in debugging | 21:27 |
| fungi | confusing symptom if so | 21:27 |
| corvus | yeah, i'm pretty sure the "ERROR" line is from ansible, and the "End" line is from the containing bash script, so that looks like an ansible bailout (there's no run summary or anything) | 21:28 |
| *** kjackal has quit IRC | 21:28 | |
| corvus | but why would that handler not be found? | 21:29 |
| fungi | i half wonder if le changed how they handle cnames for dns validation | 21:33 |
| corvus | fungi: i'm pretty sure this is an ansible problem | 21:33 |
| fungi | here's hoping | 21:34 |
| corvus | i believe that ansible aborted before running acme.sh the second time | 21:34 |
| corvus | well, | 21:34 |
| clarkb | pabelanger is the ansible handler whisperer | 21:34 |
| corvus | i mean, | 21:34 |
| corvus | i have no idea how to find out what ansible was thinking | 21:34 |
| corvus | so, i'd rather it not be an ansible problem | 21:34 |
| fungi | "Since Let’s Encrypt follows the DNS standards when looking up TXT records for DNS-01 validation, you can use CNAME records or NS records to delegate answering the challenge to other DNS zones. This can be used to delegate the _acme-challenge subdomain to a validation-specific server or zone." https://letsencrypt.org/docs/challenge-types/ | 21:36 |
| fungi | if they changed how it works, they haven't updated that document at least | 21:37 |
| corvus | fungi: i don't think anything changed. i think that's a red herring. | 21:37 |
| fungi | seems likely, yes | 21:37 |
| *** rcernin has joined #openstack-infra | 21:37 | |
| *** rcernin has quit IRC | 21:37 | |
| *** rcernin has joined #openstack-infra | 21:38 | |
| fungi | just wondering how that would have caused _acme-challenge.acme.opendev.org. to get injected in the request | 21:38 |
| corvus | fungi: let's just assume that's a bogus "helpful" message from the script and ignore it for now? | 21:39 |
| *** eharney has joined #openstack-infra | 21:42 | |
| corvus | does anyone know why some handlers use "import_tasks" and some use "include_tasks"? | 21:42 |
| corvus | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/letsencrypt-create-certs/handlers/main.yaml | 21:43 |
| clarkb | corvus: see 8a06d48c84a238903f93e779cd9dcc94bca317fd | 21:44 |
| clarkb | looks like there is a problem with import_tasks | 21:45 |
| clarkb | (thats a commit on system-config) | 21:45 |
| corvus | why wasn't tarballs updated? | 21:45 |
| corvus | or logs? | 21:45 |
| clarkb | I think tarballs was added later, perhaps there were two sets of change and that one simply didn't get updated? | 21:46 |
| corvus | so... maybe this is the problem, and it was latent until we changed something on the files02 host? | 21:46 |
| clarkb | ya | 21:46 |
| corvus | okay, i will make a stab-in-the-dark change and see if that fixes it | 21:47 |
| clarkb | ya the include switch was june 11 and the tarballs add was june 12 | 21:47 |
| clarkb | I'm going to guess those were both in flight concurrently and the change in flight didn't get updated only the existing code | 21:47 |
| rm_work | random offtopic -- spinning up a wsgi app with, for example, uwsgi -- how do I pass through stuff like the service's config file path? O_o | 21:50 |
| rm_work | https://zuul.opendev.org/t/openstack/build/f3da5dfca22d4e72933a44bddcea3989/log/controller/logs/screen-o-api.txt.gz#44 <-- like here it loads it but only because it's in the default location I think? | 21:50 |
| rm_work | figure you folks might know, ignore me if you're super busy :D | 21:50 |
| clarkb | rm_work: I think you set env vars for stuff like that | 21:51 |
| clarkb | apache lets you set env vars like that at least | 21:51 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use include_tasks in handlers https://review.opendev.org/701460 | 21:51 |
| corvus | clarkb, fungi, mordred: ^ maybe that? | 21:51 |
| fungi | lookin | 21:51 |
| clarkb | +2 | 21:51 |
| fungi | and i agree on the import_tasks vs include_tasks issue, we explicitly switched away from import_tasks so we should do this regardless | 21:52 |
| ianw_pto | thanks, fwiw i still don't think the listen: thing works, though it seems like what we really want (iirc from last time i tested it in the gate, maybe should retry with later ansibles) | 21:55 |
| rm_work | hmm apparently not from env: https://docs.openstack.org/oslo.config/latest/configuration/options.html#DEFAULT.config_file | 22:01 |
| rm_work | "This option must be set from the command-line." :( | 22:01 |
| clarkb | mordred: https://review.opendev.org/#/c/691777/ says we should update storyboard-dev to LE first, my browser says this has not yet been done | 22:02 |
| *** dpawlik has quit IRC | 22:03 | |
| mordred | clarkb: oh - I should update that commit message maybe | 22:03 |
| mordred | clarkb: I think last time we chatted we decided to just ignore that issue becuase it's not a thing on review.o.o | 22:04 |
| bnemec | rm_work: That _may_ have been written before we had an environment driver in oslo.config. | 22:04 |
| rm_work | bnemec: ah, so docs may be wrong? | 22:05 |
| clarkb | mordred: gotcha so safe to approve as is? | 22:05 |
| rm_work | it seems perplexing that there would be no way to load a correct config file, given that "wsgi app" is the recommended way to run API services (and there was a whole cycle goal around it) | 22:05 |
| mordred | clarkb: yeah | 22:05 |
| clarkb | mordred: done | 22:06 |
| bnemec | rm_work: It's possible. But config-file and config-dir are kind of bootstrap options for oslo.config so I believe they get special handling that may not include the environment driver. | 22:06 |
| mordred | clarkb: api interaction between review-dev and storyboard-dev will be wonky until we LE storyboard-dev - but I don't think we're super concerned about that at the moment? | 22:06 |
| clarkb | mordred: ya I don't think so | 22:07 |
| clarkb | mordred: also I think we are || that close to being able to LE all the things | 22:07 |
| clarkb | so if it is an issue should be easy enough to roll that out | 22:07 |
| mordred | clarkb: ++ | 22:07 |
| bnemec | rm_work: It's possible existing WSGI deployments rely on the default config file locations. | 22:09 |
| rm_work | yeah that is what it looks like O_o | 22:09 |
| *** yonglihe has quit IRC | 22:09 | |
| rm_work | this seems like a painful oversight | 22:09 |
| bnemec | It looks like oslo.config is processing cli args directly, so the environment driver wouldn't apply: https://github.com/openstack/oslo.config/blob/edcb46deae997056020acff8d24bf328778624c9/oslo_config/cfg.py#L2907 | 22:10 |
| clarkb | mordred: can you see comments in https://review.opendev.org/#/c/691800/12 | 22:10 |
| bnemec | For the WSGI use case it seems reasonable to add env var support there though. | 22:11 |
| mordred | clarkb: damn, see - I knew what those different keys were a month ago | 22:19 |
| mordred | clarkb: (very good point about documenting them) | 22:19 |
| rm_work | bnemec: can you tell if the equiv for --log-config is any different? | 22:21 |
| rm_work | that's the other file that loads up | 22:22 |
| rm_work | and i can get around the --config-file location with a workaround, but the log-config i don't think has a default location... | 22:22 |
| bnemec | rm_work: That should be fine. oslo.config doesn't need the log config in order to load files. | 22:23 |
| rm_work | but is there a way to set that with the env? | 22:23 |
| bnemec | If you have a version of oslo.config with the environment driver: https://docs.openstack.org/oslo.config/latest/reference/drivers.html#environment | 22:24 |
| rm_work | err, how would i use that | 22:24 |
| rm_work | or rather, specify to load it | 22:25 |
| rm_work | it says enabled by default but you said i need to make sure i have it? | 22:25 |
| bnemec | It's always loaded. | 22:25 |
| rm_work | is that just an age thing? | 22:25 |
| rm_work | if so, i have it :D | 22:25 |
| rm_work | (running master) | 22:25 |
| bnemec | Yeah, just the version of oslo.config. | 22:25 |
| bnemec | Perfect. :-) | 22:26 |
| rm_work | errr ok but uh | 22:26 |
| bnemec | Also, note that log-config got renamed to log-config-append: https://github.com/openstack/oslo.log/blob/7c5f8362b26313217b6c248e77be3dc8e2ef74a5/oslo_log/_options.py#L45 | 22:26 |
| rm_work | --log-config is loading up a whole file | 22:26 |
| rm_work | yeah i think that's different? | 22:26 |
| rm_work | i was looking at that | 22:26 |
| rm_work | but if you use log_config_append it then ignores `log_file` and `log_dir` in the main config, which we are using right now | 22:27 |
| *** rosmaita has quit IRC | 22:27 | |
| bnemec | Yeah, that overrides everything. If you don't want to do that you can use the other oslo.log options to customize a subset of the behavior. | 22:28 |
| bnemec | If you're using log_file and log_dir now, wouldn't you just set those in the environment instead? | 22:29 |
| rm_work | err, i mean we set those two in the main config, and use `--log-config` for other stuff, and it doesn't appear to invalidate them | 22:29 |
| rm_work | whereas if i use `log_config_append` it would | 22:29 |
| rm_work | though i guess it just moves over to the log-config file? | 22:30 |
| rm_work | i'd just add a DEFAULT section in there and add the same vars? | 22:30 |
| bnemec | Well, log-config-append is additive, so if there isn't something in your log config file it may keep whatever is set via the other options. | 22:31 |
| rm_work | hmm ok i guess i can test that | 22:31 |
| rm_work | the docs seem to indicate they'll just be ignored if i set the config-append | 22:31 |
| bnemec | I wonder if that's an oversight from when we made it append instead of replace. | 22:32 |
| rm_work | well, the env thing works, but fixing another issue before i can see if it actually worked :D | 22:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 22:36 |
| mordred | clarkb, fungi: ^^ updated for clarkb's comments | 22:36 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Update zuul-ci.org certs https://review.opendev.org/701469 | 22:36 |
| corvus | clarkb, fungi, mordred: ^ as i understand more of this, i think that's ^ actually what we need from LE | 22:36 |
| mordred | corvus: yes - that makes sense to me | 22:37 |
| bnemec | rm_work: Interesting. It looks like they should be mutually exclusive: https://github.com/openstack/oslo.log/blob/master/oslo_log/log.py#L271 | 22:41 |
| bnemec | And https://bugs.launchpad.net/oslo.log/+bug/1370439 suggests that at one point they were. | 22:41 |
| openstack | Launchpad bug 1370439 in oslo.log "log_config_append disables existing loggers" [Low,Fix released] - Assigned to Brant Knudson (blk-u) | 22:41 |
| rm_work | oh, you know what... i think they definitely are... and that whoever wrote the config i'm working on knew and was counting on it | 22:42 |
| rm_work | because the logfiles configured in the main config are for API only... and the API has a "log_config.conf" generated as well, but since it could never be used .... rofl | 22:43 |
| rm_work | meanwhile for other non-api services, the log_config would overwrite the setting for the log_file | 22:43 |
| rm_work | yeah ok, long winded way of saying i think this is fine | 22:43 |
| rm_work | (for me) | 22:43 |
| rm_work | still kinda bad that you can't specify a config loc tho | 22:44 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use LE certs for zuul-ci.org https://review.opendev.org/701472 | 22:46 |
| bnemec | I suspect it wouldn't be that difficult to add env var support for config-file and config-dir. We could follow the same pattern used for the environment driver. | 22:46 |
| corvus | i think that's the end state once the certs are all issued ^ | 22:46 |
| *** pkopec has quit IRC | 22:49 | |
| corvus | clarkb, mordred, fungi: do you understand the job failures on https://review.opendev.org/700083 ? | 22:52 |
| corvus | those don't seem to be failures of the new test i wrote. | 22:52 |
| corvus | oh i guess only one of those jobs is voting | 22:53 |
| mordred | corvus: I'm confused by your commit message on 701472 | 22:53 |
| corvus | so https://zuul.opendev.org/t/openstack/build/b37308845e9f48d98da7fa3b5833b798 | 22:53 |
| clarkb | corvus: https://zuul.opendev.org/t/openstack/build/b37308845e9f48d98da7fa3b5833b798/log/logs/centos-minimal_8-build-succeeds.FAIL.log#988 | 22:53 |
| clarkb | problem with centos 8 mirror downloads in ovh land? | 22:54 |
| corvus | clarkb: aha, thanks. that's an old result too, so maybe just ignore and recheck. | 22:54 |
| clarkb | sounds good | 22:54 |
| mordred | ++ | 22:54 |
| corvus | clarkb: i saw a bunch of failures and assumed something bigger was wrong, but in looking at other changes, i think the 2 failures are expected, and that one was just bad luck. | 22:54 |
| corvus | however, i think i may have missed adding my test to the test run, so i'll just push up a new ps. | 22:55 |
| corvus | mordred: how can i clarify commit msg? | 22:55 |
| mordred | corvus: in 701472 you say the the old vhost doesn't support certs by files so you're goign to use the new "website" method - but both the old and the new things are using openstack_project::website | 22:55 |
| mordred | so I don't know that I'm following what you mean - I might well just be missing something obvious | 22:56 |
| corvus | mordred: see manifests/files.pp lines 335-369. those are removed and replaced with site.pp lines 491-497 | 22:56 |
| mordred | ahhhhhhhh | 22:56 |
| mordred | thanks | 22:56 |
| clarkb | corvus: mordred hrm looking at that again I think we may not be able to do that replacement quite like that | 22:56 |
| mordred | I was fixated on the one website -> two websites in the same file | 22:56 |
| clarkb | the website stuff comes with a templated vhost for serving stuff off afs | 22:57 |
| corvus | mordred: ah yep. it's website->website and vhost->websit | 22:57 |
| clarkb | but the redirect would just be a redirect I think | 22:57 |
| corvus | clarkb: if i followed it correctly, supplying a docroot overrides that | 22:57 |
| corvus | please feel free to double check me | 22:57 |
| clarkb | corvus: it override the docroot value (so not serving out of afs) but I don't think it will use the different vhost template which includes the redirect rules | 22:58 |
| clarkb | if we switched to using .htpassword file redirects we could do it this way and put that file in the docroot | 22:58 |
| fungi | which redirects would we want to do that with? | 22:59 |
| clarkb | fungi: the git.zuul-ci.org -> opendev.org redirects | 22:59 |
| fungi | if it's redirecting from specific domains, i vaguely recall .htpassword can't match on the domain part | 23:00 |
| corvus | i thought we did use .htaccess redirects | 23:00 |
| corvus | (i assume that's what you meant) | 23:00 |
| fungi | though maybe that was just with older apache | 23:00 |
| clarkb | er ya .htaccess | 23:00 |
| clarkb | corvus: I'm trying to pull that up to confirm now | 23:00 |
| fungi | oh, .htaccess yes | 23:00 |
| fungi | finger memory | 23:00 |
| * fungi sighs | 23:00 | |
| openstackgerrit | Merged opendev/system-config master: Plumb through storyboard hiera data https://review.opendev.org/691777 | 23:01 |
| openstackgerrit | Merged opendev/puppet-zuul master: Fix linter errors https://review.opendev.org/701278 | 23:01 |
| corvus | it does look like maybe we have an extended AllowOverrideList | 23:01 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/git-redirect.vhost.erb | 23:01 |
| clarkb | ok I think you are right | 23:01 |
| clarkb | the redirects are in the docroot and not in the vhost | 23:01 |
| clarkb | this change should be fine as is | 23:01 |
| corvus | well, maybe not because of the allowoverride | 23:02 |
| clarkb | oh | 23:02 |
| bnemec | rm_work: I opened an RFE for the env var support: https://bugs.launchpad.net/oslo.config/+bug/1858708 | 23:02 |
| openstack | Launchpad bug 1858708 in oslo.config "RFE: Support setting config-file and config-dir from environment variables" [Wishlist,Triaged] | 23:02 |
| rm_work | bnemec: oh, thanks! :) | 23:02 |
| clarkb | corvus: maybe what we want is to addin in the ability to pass the vhost into website | 23:02 |
| fungi | ahh, it does appear that %{HTTP_HOST} is usable for RewriteCond matches in .htaccess files these days | 23:03 |
| clarkb | corvus: then we can set docroot and vhost override both there | 23:03 |
| fungi | at least based on some examples i'm finding | 23:03 |
| corvus | clarkb: yeah, might be the simplest solution | 23:03 |
| fungi | so if we want to do the git.zuul-ci.org redirect in a .htaccess file, we should be able to | 23:03 |
| corvus | fungi: it's already there https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git-redirect.htaccess | 23:05 |
| *** laurenceee has joined #openstack-infra | 23:05 | |
| fungi | oh neat | 23:07 |
| fungi | right, and now i remember why ;) | 23:08 |
| laurenceee | I pushed a tag to a review.openstack.org (openstack/networking-calico) but it hasn't shown up in https://opendev.org/openstack/networking-calico.git | 23:08 |
| *** tkajinam has joined #openstack-infra | 23:08 | |
| *** rlandy is now known as rlandy|bbl | 23:08 | |
| laurenceee | Is there a delay in the synchronization or am I missing something else :) | 23:08 |
| clarkb | laurenceee: there can be a slight delay. what tag did you push? | 23:09 |
| laurenceee | clarkb: 3.9.5 | 23:09 |
| fungi | corvus: even more ironic is the fact that i'm the one who added that file (adapted from your version for git.openstack.org) | 23:09 |
| fungi | clearly it's getting late here | 23:09 |
| clarkb | laurenceee: https://opendev.org/openstack/networking-calico/src/tag/3.9.5 it is there for me (at least on the backend I am hitting) | 23:10 |
| clarkb | laurenceee: let me check all 8 backends | 23:10 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: Add containerfile element https://review.opendev.org/700083 | 23:10 |
| clarkb | https://gitea08.opendev.org:3000/openstack/networking-calico/src/tag/3.9.5 is a 404 | 23:11 |
| laurenceee | clarkb: thanks. Yeah, that's what I'm seeing: 404's | 23:11 |
| clarkb | laurenceee: if you inspect the ssl cert for opendev.org it will tell you which one you are hitting, can you confirm it is gitea08 too? | 23:11 |
| clarkb | my hunch is the restarts for the 1.10.2 upgrade left a stale git config lock file and replication to 08 is not working | 23:11 |
| laurenceee | clarkb: yep, that's what the TLS cert says too: gitea08 | 23:12 |
| fungi | there are no outstanding replication tasks in gerrit's queue | 23:12 |
| *** hamzy has quit IRC | 23:13 | |
| fungi | alternate theory, could gitea08's git ssh interface have been down for a restart the moment gerrit tried to replicate? | 23:13 |
| *** tesseract has quit IRC | 23:13 | |
| clarkb | fungi: that could be too | 23:13 |
| clarkb | I'm working on checking the lock file now | 23:14 |
| corvus | clarkb, fungi: the website template does a http->https redirect, but the git-redirect template does not. think that's an okay change? | 23:14 |
| fungi | corvus: seems like it should be fine to add to the .htaccess file, i have a hard time believing the protocol isn't included in the envvars .htaccess rules can see | 23:15 |
| corvus | fungi: no, i mean the vhost template | 23:16 |
| *** goldyfruit_ has joined #openstack-infra | 23:17 | |
| fungi | oh | 23:17 |
| corvus | the operative question is: currently a user who visits "http://git.zuul-ci.org/foo" is redirected to "https://opendev.org/foo"; is it okay to change that to http://git.zuul-ci.org/foo -> https://git.zuul-ci.org/foo -> https://opendev.org/foo | 23:17 |
| corvus | ie, add an extra silly redirect for http(non-s) users | 23:18 |
| fungi | yeah, that's also likely fine as long as it will still match the .htaccess rules (it should) | 23:19 |
| laurenceee | (Thanks for the help btw, I appreciate it. I'll use one of the other gitea hosts for now :) ) | 23:19 |
| mordred | I also think the silly redirect is fine | 23:19 |
| clarkb | laurenceee: its there now. | 23:20 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use LE certs for zuul-ci.org https://review.opendev.org/701472 | 23:20 |
| clarkb | fungi: I think you were right simply triggering replication was enough to get it there | 23:20 |
| clarkb | I'm going to trigger a full replication against 08 now | 23:20 |
| fungi | corvus: going back over the .htaccess file i don't think it should cause a problem, we don't match on the protocol anywhere | 23:20 |
| laurenceee | clarkb: Awesome, thanks! | 23:20 |
| corvus | fungi: yes, it was intended to work for both in one step | 23:20 |
| fungi | clarkb: yeah, the tag was created at 21:30:45 utc so i was about to find the push in the gerrit ssh log to get a better guess as to when it would have tried to replicate | 23:21 |
| *** aaronsheffield has quit IRC | 23:21 | |
| corvus | gerrit knows when replication fails, i wonder why it didn't retry? | 23:22 |
| fungi | there's a push from what looks like laurenceee's user to that repo at 2020-01-07 21:30:59,983 | 23:22 |
| fungi | so guessing that's the tag | 23:23 |
| clarkb | corvus: re 472 is that scope lookupvar the way to lookup a define specific var? (I don't know) | 23:23 |
| clarkb | cmurphy: ^ maybe you know | 23:23 |
| fungi | corvus: i don't see in gerrit's error log where it even admitted to a replication failure around that timeframe :/ | 23:24 |
| openstackgerrit | Merged opendev/system-config master: Use include_tasks in handlers https://review.opendev.org/701460 | 23:25 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use LE certs for zuul-ci.org https://review.opendev.org/701472 | 23:25 |
| fungi | corvus: i take that back, i should have been looking in the replication log, that's apparently separate | 23:26 |
| *** rh-jelabarre has quit IRC | 23:26 | |
| openstackgerrit | Merged opendev/puppet-zuul master: Add option for LE provisioned certs to zuul::web https://review.opendev.org/701265 | 23:27 |
| clarkb | https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser/scope.rb#L481-L513 currently trying to understand that | 23:28 |
| fungi | corvus: that said, it logs the push of refs/tags/3.9.5 to gitea08 but doesn't log an error | 23:29 |
| fungi | still, logged that it pushed at 2020-01-07 21:31:00,983 so i'll see if i can match that to a restart | 23:29 |
| cmurphy | clarkb: my puppet is rusty but i think latest patchset is right, the template is being evaluated by the template() function in website.pp so allow_override_list should be available | 23:29 |
| clarkb | cmurphy: aha thanks | 23:30 |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| *** dchen has joined #openstack-infra | 23:32 | |
| clarkb | fungi: corvus: could that be a problem syncing the git process details out to disk racing against the container stopping? typically though when a process stops it flushes buffers to disk right? | 23:33 |
| fungi | fungi@gitea08:~$ sudo docker exec -t giteadocker_gitea-ssh_1 ls -l /var/run/sshd.pid | 23:34 |
| fungi | -rw-r--r-- 1 root root 2 Jan 7 21:30 /var/run/sshd.pid | 23:34 |
| fungi | actually, stat is more specific, gives 2020-01-07 21:30:07.470763665 | 23:34 |
| *** ociuhandu has quit IRC | 23:35 | |
| fungi | so the push was roughly 52.5 seconds after the sshd pidfile was created. maybe clocks somewhere aren't accurate, or maybe the sshd actually takes that long to be usable? | 23:35 |
| *** aedc has quit IRC | 23:36 | |
| fungi | either way, the container restart was within a minute of the replication push, so this seems likely to be related | 23:36 |
| clarkb | the gitea restarts actually do take a little bit of time iirc | 23:36 |
| clarkb | but not sure where in that process the time is lost | 23:37 |
| fungi | well, regardless, i think it's safe to say that if we restart gitea containers (e.g., for upgrades) we may need force replication in gerrit | 23:37 |
| clarkb | or force it to error so that gerrit retries | 23:38 |
| clarkb | oh you know | 23:38 |
| clarkb | I wonder if this is actually a case of the git config lockfile? | 23:38 |
| fungi | oh, and gets cleared as part of the startup? | 23:38 |
| fungi | just delayed? | 23:39 |
| clarkb | ya | 23:39 |
| clarkb | perhaps the lock is held until gitea finishes starting itself if it was previous locked? | 23:39 |
| clarkb | and intermediate git processes "succeed" like we've seen in the past | 23:39 |
| fungi | well, regardless, if there's a window of roughly a minute (maybe longer, who knows?) where each of the backends is not getting commits pushed for replication, then there's probably plenty more missing from the restart besides just that tag on a single backend | 23:39 |
| clarkb | yup I'm going to queue up replication for the other 7 backends | 23:39 |
| fungi | thanks! | 23:40 |
| fungi | it's leading me to wonder whether our gitea upgrade playbook should also trigger a full replication in gerrit | 23:41 |
| *** bdodd has quit IRC | 23:41 | |
| *** mattw4 has quit IRC | 23:45 | |
| openstackgerrit | Merged opendev/system-config master: Update zuul-ci.org certs https://review.opendev.org/701469 | 23:48 |
| *** ahosam has joined #openstack-infra | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!