| clarkb | fwiw I've not had a chance to look at puppet yet but will try a stab at fixing it for 2.13.12 upgrade tomorrow and hopefully thats athis week thibg. Then we ahould totally look at review-dev doing the 2.15 docker stuff | 00:01 |
|---|---|---|
| mordred | clarkb: ++ | 00:01 |
| mordred | clarkb: so - actually, .I had this idea, that we take the wars we have built for 2.13.12, build a docker image with them, then do a switch to docker but sticking with 2.13.12 - then do the 2.15 upgrade ... that way for the 2.15 upgrade we're only changing one thing, and same for the docker | 00:02 |
| clarkb | mordred also I want nick foles to win the superb owl this year for the memes | 00:02 |
| clarkb | mordred: oh I like that | 00:03 |
| mordred | clarkb: I think we can do it with a simple one-off image - like a dockerfile that's just wgetting the existing files or something | 00:03 |
| clarkb | ya | 00:04 |
| mordred | clarkb: if we don't watch out - we're going to find ourselves running 2.16 before 2.17 is released | 00:05 |
| mordred | clarkb: if foles beats the sains, I'll definitely be rooting for them | 00:06 |
| mordred | just because the whole foles story is epic craziness | 00:06 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** yamamoto has joined #openstack-infra | 00:18 | |
| *** jamesmcarthur has joined #openstack-infra | 00:22 | |
| *** anteaya has joined #openstack-infra | 00:22 | |
| *** tosky has quit IRC | 00:22 | |
| *** yamamoto has quit IRC | 00:23 | |
| *** anteaya has left #openstack-infra | 00:26 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** anteaya has joined #openstack-infra | 00:27 | |
| *** jamesdenton has quit IRC | 00:57 | |
| *** jamesdenton has joined #openstack-infra | 00:57 | |
| *** armax has quit IRC | 01:00 | |
| *** markvoelker has joined #openstack-infra | 01:09 | |
| *** yamamoto has joined #openstack-infra | 01:13 | |
| *** yamamoto has quit IRC | 01:17 | |
| *** jamesmcarthur has joined #openstack-infra | 01:23 | |
| *** jamesmcarthur has quit IRC | 01:28 | |
| *** hwoarang has quit IRC | 01:32 | |
| *** hwoarang has joined #openstack-infra | 01:32 | |
| fungi | mordred: wouldn't just lose review history, but also things like project watches. honestly there's not much point to the importer plugin for us, i don't think, unless we want to use it to replace some of jeepyb's manage-projects | 01:40 |
| fungi | clarkb: ^ | 01:40 |
| mordred | fungi: yeah. I think you're right | 01:41 |
| fungi | (which might still be a good thing to check into) | 01:41 |
| mordred | oh jeez: http://logs.openstack.org/68/630468/9/check/system-config-build-image-gerrit/987a974/job-output.txt.gz#_2019-01-13_23_54_25_979978 | 01:41 |
| mordred | "There is insufficient memory for the Java Runtime Environment to continue." | 01:42 |
| fungi | hopefully that can be tuned? | 01:42 |
| mordred | hopefully. although maybe it's a secondary error due to http://logs.openstack.org/68/630468/9/check/system-config-build-image-gerrit/987a974/job-output.txt.gz#_2019-01-13_23_54_25_975151 | 01:43 |
| mordred | this built on my laptop yesterday :( | 01:43 |
| *** markvoelker has quit IRC | 01:45 | |
| *** yamamoto has joined #openstack-infra | 01:48 | |
| *** yamamoto has quit IRC | 01:55 | |
| *** hwoarang has quit IRC | 01:57 | |
| *** jamesmcarthur has joined #openstack-infra | 01:58 | |
| *** hwoarang has joined #openstack-infra | 01:58 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** hongbin has joined #openstack-infra | 02:14 | |
| *** slaweq has quit IRC | 02:15 | |
| *** ruffian_sheep has joined #openstack-infra | 02:27 | |
| *** armax has joined #openstack-infra | 02:33 | |
| *** yamamoto has joined #openstack-infra | 02:43 | |
| *** armax has quit IRC | 02:46 | |
| *** yamamoto has quit IRC | 02:47 | |
| *** psachin has joined #openstack-infra | 02:57 | |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 02:57 |
| *** jamesmcarthur has joined #openstack-infra | 02:58 | |
| *** remi_ness has quit IRC | 03:01 | |
| *** jamesmcarthur has quit IRC | 03:03 | |
| *** remi_ness has joined #openstack-infra | 03:15 | |
| *** markvoelker has joined #openstack-infra | 03:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: add playbooks to job.toDict() https://review.openstack.org/621343 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Get executor job params https://review.openstack.org/607078 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Separate out executor server from runner https://review.openstack.org/607079 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tests: improve test_web to only provision events when needed https://review.openstack.org/630575 | 03:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 03:22 |
| *** yamamoto has joined #openstack-infra | 03:23 | |
| *** yamamoto has quit IRC | 03:28 | |
| *** whoami-rajat has joined #openstack-infra | 03:30 | |
| *** markvoelker has quit IRC | 03:34 | |
| *** markvoelker has joined #openstack-infra | 03:34 | |
| *** markvoelker has quit IRC | 03:39 | |
| *** bobh has joined #openstack-infra | 03:40 | |
| *** diablo_rojo has joined #openstack-infra | 03:41 | |
| *** markvoelker has joined #openstack-infra | 03:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tests: improve test_web to only provision events when needed https://review.openstack.org/630575 | 03:49 |
| *** markvoelker has quit IRC | 03:54 | |
| *** markvoelker has joined #openstack-infra | 03:55 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** bobh has quit IRC | 04:01 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** hwoarang has quit IRC | 04:08 | |
| *** hwoarang has joined #openstack-infra | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** ramishra has joined #openstack-infra | 04:16 | |
| *** slaweq has quit IRC | 04:16 | |
| *** yamamoto has joined #openstack-infra | 04:17 | |
| *** yamamoto has quit IRC | 04:21 | |
| *** udesale has joined #openstack-infra | 04:23 | |
| *** hwoarang has quit IRC | 04:29 | |
| *** hwoarang has joined #openstack-infra | 04:30 | |
| *** owalsh_ has joined #openstack-infra | 04:33 | |
| *** owalsh has quit IRC | 04:37 | |
| *** yamamoto has joined #openstack-infra | 04:50 | |
| *** wolverineav has joined #openstack-infra | 04:55 | |
| *** yamamoto has quit IRC | 04:56 | |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| *** jamesmcarthur has quit IRC | 05:05 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tests: improve test_web to only provision events when needed https://review.openstack.org/630575 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: add playbooks to job.toDict() https://review.openstack.org/621343 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Get executor job params https://review.openstack.org/607078 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Separate out executor server from runner https://review.openstack.org/607079 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Move common AnsibleJob prep tasks into a base class https://review.openstack.org/607080 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Implement a local zuul-runner https://review.openstack.org/607082 | 05:19 |
| *** markvoelker has quit IRC | 05:39 | |
| *** jamesmcarthur has joined #openstack-infra | 05:41 | |
| *** yamamoto has joined #openstack-infra | 05:44 | |
| *** jamesmcarthur has quit IRC | 05:46 | |
| *** yamamoto has quit IRC | 05:49 | |
| *** wolverineav has quit IRC | 05:51 | |
| *** wolverineav has joined #openstack-infra | 05:55 | |
| *** hongbin_ has joined #openstack-infra | 05:59 | |
| *** diablo_rojo has quit IRC | 06:00 | |
| *** hongbin has quit IRC | 06:02 | |
| *** hongbin_ has quit IRC | 06:02 | |
| *** wolverineav has quit IRC | 06:07 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** udesale has quit IRC | 06:11 | |
| *** udesale has joined #openstack-infra | 06:12 | |
| *** markvoelker has joined #openstack-infra | 06:12 | |
| *** slaweq has quit IRC | 06:15 | |
| *** ykarel has joined #openstack-infra | 06:19 | |
| *** e0ne has joined #openstack-infra | 06:19 | |
| *** ruffian_sheep has quit IRC | 06:26 | |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 06:33 |
| *** wolverineav has joined #openstack-infra | 06:41 | |
| *** wolverineav has quit IRC | 06:45 | |
| *** slaweq has joined #openstack-infra | 06:46 | |
| *** jbadiapa has joined #openstack-infra | 06:46 | |
| *** slaweq has quit IRC | 06:50 | |
| *** remi_ness has quit IRC | 06:51 | |
| *** e0ne has quit IRC | 06:57 | |
| *** udesale has quit IRC | 06:58 | |
| *** udesale has joined #openstack-infra | 06:58 | |
| *** yamamoto has joined #openstack-infra | 07:05 | |
| *** slaweq has joined #openstack-infra | 07:06 | |
| *** yamamoto_ has joined #openstack-infra | 07:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 07:08 |
| *** yamamoto has quit IRC | 07:11 | |
| *** pcaruana has joined #openstack-infra | 07:11 | |
| *** hwoarang has quit IRC | 07:13 | |
| *** wolverineav has joined #openstack-infra | 07:13 | |
| *** hwoarang has joined #openstack-infra | 07:14 | |
| *** slaweq has quit IRC | 07:15 | |
| *** bdodd has quit IRC | 07:15 | |
| *** kjackal has joined #openstack-infra | 07:22 | |
| *** yamamoto_ has quit IRC | 07:22 | |
| *** Dobroslaw has joined #openstack-infra | 07:25 | |
| *** dpawlik has joined #openstack-infra | 07:29 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Move common AnsibleJob prep tasks into a base class https://review.openstack.org/607080 | 07:31 |
| *** jtomasek has joined #openstack-infra | 07:36 | |
| *** rascasoft has joined #openstack-infra | 07:37 | |
| *** kjackal has quit IRC | 07:42 | |
| *** ccamacho has joined #openstack-infra | 07:42 | |
| *** jamesmcarthur has joined #openstack-infra | 07:42 | |
| *** jamesmcarthur has quit IRC | 07:47 | |
| *** kopecmartin has joined #openstack-infra | 07:52 | |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 07:58 |
| *** markvoelker has quit IRC | 07:58 | |
| *** slaweq has joined #openstack-infra | 07:58 | |
| *** lpetrut has joined #openstack-infra | 08:01 | |
| *** yamamoto has joined #openstack-infra | 08:03 | |
| *** markvoelker has joined #openstack-infra | 08:03 | |
| *** gfidente has joined #openstack-infra | 08:05 | |
| *** rossella_s has quit IRC | 08:05 | |
| *** wolverineav has quit IRC | 08:08 | |
| *** yamamoto has quit IRC | 08:08 | |
| *** rpittau has joined #openstack-infra | 08:09 | |
| *** markvoelker has quit IRC | 08:10 | |
| *** ginopc has joined #openstack-infra | 08:11 | |
| *** rossella_s has joined #openstack-infra | 08:12 | |
| *** bdodd has joined #openstack-infra | 08:22 | |
| *** pgaxatte has joined #openstack-infra | 08:26 | |
| *** ginopc has quit IRC | 08:26 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add action to task result in zuul_json callback https://review.openstack.org/630622 | 08:28 |
| *** markvoelker has joined #openstack-infra | 08:29 | |
| *** pfallenop has quit IRC | 08:30 | |
| *** markvoelker has quit IRC | 08:30 | |
| *** markvoelker has joined #openstack-infra | 08:32 | |
| *** markvoelker has quit IRC | 08:33 | |
| openstackgerrit | Tobias Urdin proposed openstack-infra/project-config master: Add puppetforge publish and add to puppet release https://review.openstack.org/627573 | 08:34 |
| *** markvoelker has joined #openstack-infra | 08:35 | |
| *** tosky has joined #openstack-infra | 08:37 | |
| *** jangutter has joined #openstack-infra | 08:37 | |
| *** pfallenop has joined #openstack-infra | 08:41 | |
| *** mudpuppy has quit IRC | 08:47 | |
| *** mudpuppy has joined #openstack-infra | 08:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 08:51 |
| *** ginopc has joined #openstack-infra | 08:53 | |
| *** ssbarnea|rover has joined #openstack-infra | 08:54 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add action to task result in zuul_json callback https://review.openstack.org/630622 | 08:54 |
| *** jpich has joined #openstack-infra | 08:56 | |
| *** jpena|off is now known as jpena | 08:58 | |
| *** roman_g has joined #openstack-infra | 08:58 | |
| *** panda|off is now known as panda | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** yamamoto has quit IRC | 09:21 | |
| *** chandan_kumar is now known as chandankumar | 09:21 | |
| *** apetrich has joined #openstack-infra | 09:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 09:27 |
| *** derekh has joined #openstack-infra | 09:32 | |
| *** owalsh_ is now known as owalsh | 09:34 | |
| *** kjackal has joined #openstack-infra | 09:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Implement a local zuul-runner https://review.openstack.org/607082 | 09:40 |
| *** markvoelker has quit IRC | 09:52 | |
| *** wolverineav has joined #openstack-infra | 09:55 | |
| *** kjackal has quit IRC | 09:55 | |
| *** kjackal has joined #openstack-infra | 09:56 | |
| *** wolverineav has quit IRC | 09:59 | |
| tobias-urdin | infra-root requesting review on https://review.openstack.org/#/c/627553/ and https://review.openstack.org/#/c/627573/ we just merged stein-2 milestone so not sure if there is anything we could test releasing right now | 10:00 |
| tobias-urdin | i can probably come up with a module that we can do a release with | 10:00 |
| *** kjackal has quit IRC | 10:04 | |
| *** e0ne has joined #openstack-infra | 10:07 | |
| *** sshnaidm|off is now known as sshnaidm | 10:27 | |
| *** d0ugal has quit IRC | 10:29 | |
| frickler | amorin: looking at the last 7 days for grafana, we seem to be capped at about 120 nodes in bhs1 instead of the configure maximum of 150, can you please double-check our quota there? grafana.openstack.org/d/rZtIH5Imz/nodepool | 10:33 |
| *** dtantsur|afk is now known as dtantsur | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** electrofelix has joined #openstack-infra | 10:45 | |
| *** roman_g has quit IRC | 10:46 | |
| *** roman_g_ has joined #openstack-infra | 10:46 | |
| *** d0ugal has joined #openstack-infra | 10:48 | |
| *** yamamoto has quit IRC | 10:49 | |
| *** udesale has quit IRC | 11:13 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 11:19 |
| openstackgerrit | Tobias Urdin proposed openstack-infra/zuul-jobs master: Add upload-puppetforge role https://review.openstack.org/627553 | 11:30 |
| openstackgerrit | Tobias Urdin proposed openstack-infra/project-config master: Add puppetforge publish and add to puppet release https://review.openstack.org/627573 | 11:32 |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** rfolco has joined #openstack-infra | 11:38 | |
| *** yamamoto has quit IRC | 11:41 | |
| *** roman_g_ has quit IRC | 11:47 | |
| *** finucannot is now known as stephenfin | 11:49 | |
| *** roman_g has joined #openstack-infra | 11:53 | |
| dpawlik | Hi, does Openstack documentation have any information, which cpu_model_extra_flags are "recommended" to enable? | 11:54 |
| *** yamamoto has joined #openstack-infra | 11:59 | |
| *** ykarel is now known as ykarel|away | 12:00 | |
| *** roman_g has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-infra | 12:01 | |
| *** roman_g has joined #openstack-infra | 12:02 | |
| *** Qianbiao has joined #openstack-infra | 12:05 | |
| Qianbiao | ironic | 12:06 |
| *** ykarel|away has quit IRC | 12:07 | |
| *** Qianbiao has quit IRC | 12:13 | |
| lucasagomes | hi all, I'm trying to use a reproducer.sh from gate (http://logs.openstack.org/54/625554/9/check/networking-ovn-dsvm-functional/942c883/logs/reproduce.sh) but even following all the instructions on the script (see comments at the top) it fails with "/usr/zuul-env/bin/zuul-cloner: No such file or directory" | 12:20 |
| lucasagomes | anyone has seen it ? /me googles to see how to get zuul-cloner installed in the meantime | 12:21 |
| *** yamamoto has quit IRC | 12:22 | |
| *** dkehn has quit IRC | 12:25 | |
| *** jpena is now known as jpena|lunch | 12:27 | |
| *** yamamoto has joined #openstack-infra | 12:30 | |
| cmurphy | lucasagomes: you can use this script to build an image that has zuul-cloner and everything else installed http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh | 12:33 |
| *** e0ne has quit IRC | 12:33 | |
| lucasagomes | cmurphy, oh thanks for the tip. I will take a look | 12:34 |
| *** yamamoto has quit IRC | 12:34 | |
| *** quiquell is now known as quiquell|lunch | 12:48 | |
| lucasagomes | cmurphy, btw, where can I download the image(s) that are used by zuul in the gate ? | 12:53 |
| cmurphy | lucasagomes: it's not possible to download them, best you can do is use that script to generate an identical image | 12:53 |
| lucasagomes | cause I can't reproduce the error I am seeing locally, so I think it may have something to do with the version of the OS being used and so on | 12:54 |
| lucasagomes | cmurphy, gotcha... ok thanks! | 12:54 |
| *** rlandy has joined #openstack-infra | 12:58 | |
| *** boden has joined #openstack-infra | 12:59 | |
| *** rpittau is now known as rpittau|lunch | 12:59 | |
| *** needssleep is now known as TheJulia | 13:02 | |
| *** jcoufal has joined #openstack-infra | 13:03 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Adds JAVA_HOME to tox passenv https://review.openstack.org/630661 | 13:05 |
| *** quiquell|lunch is now known as quiquell | 13:05 | |
| *** ykarel has joined #openstack-infra | 13:05 | |
| *** e0ne has joined #openstack-infra | 13:18 | |
| frickler | lucasagomes: cmurphy: yes, reproduce.sh doesn't work anymore for quite some time, I proposed to remove it to avoid confusion, but that patch has gotten stale in the meantime, too https://review.openstack.org/572367 | 13:29 |
| *** dkehn has joined #openstack-infra | 13:31 | |
| *** jpena|lunch is now known as jpena | 13:31 | |
| *** wolverineav has joined #openstack-infra | 13:31 | |
| frickler | lucasagomes: if you want a node to be held so that you can look at the failing setup on a node directly in gate, just ping infra-root | 13:32 |
| lucasagomes | frickler, thanks a lot for the pointers, I just commented on that patch with my experience (hoping to bump it) | 13:34 |
| lucasagomes | frickler, I'm trying some things and generating an image with the script cmurphy pointed me to in parallel | 13:34 |
| lucasagomes | if I can't reproduce still I will try to ping infra-root to see if I can troubleshoot a VM there | 13:35 |
| *** wolverineav has quit IRC | 13:36 | |
| *** adriancz has joined #openstack-infra | 13:37 | |
| lucasagomes | frickler, fyi, that patch has a merge conflict apparently ("Cannot Merge") | 13:38 |
| lucasagomes | u may want to rebase it | 13:38 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Stop creating reproduce.sh https://review.openstack.org/572367 | 13:43 |
| frickler | lucasagomes: yeah, I assumed nobody was interested in it anyway. rebased now | 13:44 |
| *** mriedem has joined #openstack-infra | 13:44 | |
| lucasagomes | thanks, I've re-added my vote | 13:44 |
| *** kjackal has joined #openstack-infra | 13:46 | |
| fungi | lucasagomes: cmurphy: our images are actually downloadable from https://nb01.openstack.org/images/ you just need to accept the self-signed snakeoil cert there. fair warning though, booting them isn't entirely straightforward. you'll need to mount a configdrive with ssh keys you want to use to log in | 13:49 |
| *** diablo_rojo has joined #openstack-infra | 13:49 | |
| fungi | (and if you need arm64/aarch64 images, use nb03 instead of nb01) | 13:50 |
| lucasagomes | fungi, oh thanks... I can trythat if the build image does work (it's running atm) | 13:51 |
| lucasagomes | I can probably inject the keys with guestfish | 13:51 |
| fungi | but yeah, we try to make our images easy to build on your own so you can modify them in ways which make them more convenient for local use | 13:52 |
| *** sthussey has joined #openstack-infra | 13:53 | |
| lucasagomes | fair enough... this is all an attempt to reproduce the problem I'm seeing in gate since I wasn't able so far using a centos machine I had available here | 13:53 |
| lucasagomes | thanks all for the tips/inputs! | 13:53 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Document default values of runtime arguments https://review.openstack.org/630679 | 13:59 |
| *** kgiusti has joined #openstack-infra | 13:59 | |
| *** rpittau|lunch is now known as rpittau | 14:05 | |
| *** yboaron has joined #openstack-infra | 14:06 | |
| *** e0ne has quit IRC | 14:12 | |
| *** diablo_rojo has quit IRC | 14:13 | |
| *** diablo_rojo_ has joined #openstack-infra | 14:13 | |
| *** diablo_rojo has joined #openstack-infra | 14:13 | |
| *** diablo_rojo has quit IRC | 14:13 | |
| *** diablo_rojo_ has quit IRC | 14:13 | |
| *** beekneemech is now known as bnemec | 14:19 | |
| ssbarnea|rover | fungi: i am curious if you have prev knowledge regarding speed of rsync w/ o w/o compress option. if it could speedup collection of logs or not really. | 14:19 |
| *** e0ne has joined #openstack-infra | 14:26 | |
| *** markvoelker has quit IRC | 14:31 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/irc-meetings master: Update kuryr-kubernetes irc meeting time https://review.openstack.org/630689 | 14:33 |
| *** markvoelker has joined #openstack-infra | 14:34 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul-jobs master: WIP: Default private_ipv4 to use public_ipv4 address when null https://review.openstack.org/623294 | 14:36 |
| *** e0ne has quit IRC | 14:40 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 14:42 |
| *** pabelanger has joined #openstack-infra | 14:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 14:44 |
| fungi | ssbarnea|rover: it really depends on the circumstances. it's a cpu (on the sending side mainly) usage vs bandwidth consumption | 14:44 |
| fungi | ssbarnea|rover: i tend to be more worried about bandwidth usage on the zuul executors since everything's funneled through them | 14:47 |
| fungi | but cacti will tell us how bad that gets | 14:47 |
| ssbarnea|rover | fungi: i know. this is why I asked if anyone made some testing on this. i am not sure what is the major bottleneck most of the time cpu or networking. | 14:48 |
| ssbarnea|rover | i can easily do benchmarks locally but this would not tell me the true story about what happens on our CI. | 14:48 |
| fungi | looks like we spike up around 20mbps on ze01 at peak these days http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64007&rra_id=all | 14:49 |
| fungi | so not too bad | 14:49 |
| fungi | i think we get up to 100mbps in+out with that flavor but would need to double-check | 14:49 |
| *** e0ne has joined #openstack-infra | 14:50 | |
| fungi | whatever rackspace's 8gb ram standard pvhvm flavor gets anyway | 14:50 |
| *** e0ne has quit IRC | 14:53 | |
| *** e0ne has joined #openstack-infra | 14:54 | |
| *** psachin has quit IRC | 14:56 | |
| cmurphy | fungi: oh good to know re downloading images | 14:56 |
| fungi | ssbarnea|rover: i think what we've observed in the past is that delays for log collection are related less to data compression and transfer time, and impacted much more by indexing massive numbers of files, so if it's a job that collects many, many (thousands of?) files, it could be that | 14:58 |
| *** ekultails has joined #openstack-infra | 15:00 | |
| ssbarnea|rover | fungi: based on my local tests, adding "-z" to rsync doubled my rsync speed. (-z not -c, -c increased duration ~20). I am going to make a test on tripleo as ½ speed is not something to ignore on a task that takes 15 min on average. | 15:01 |
| ssbarnea|rover | fungi: and thanks for the insights. | 15:02 |
| fungi | you bet | 15:02 |
| fungi | glad to help | 15:02 |
| corvus | ssbarnea|rover: mordred is working on a change to move log fetching into the base job. if you like the results of the test and want to change, you may want to target using this system. here's the role: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/fetch-output/tasks/main.yaml | 15:06 |
| *** ykarel is now known as ykarel|away | 15:06 | |
| ssbarnea|rover | corvus: sure. still first step is to validate my theory with two specific tasks that I know about. after this we can think about scaling the solution. | 15:07 |
| corvus | sounds good | 15:08 |
| *** derekh has quit IRC | 15:11 | |
| *** dpawlik has quit IRC | 15:11 | |
| *** jangutter has quit IRC | 15:12 | |
| *** e0ne_ has joined #openstack-infra | 15:13 | |
| *** e0ne has quit IRC | 15:14 | |
| *** xek has joined #openstack-infra | 15:15 | |
| *** sreejithp has joined #openstack-infra | 15:15 | |
| *** markvoelker has quit IRC | 15:20 | |
| *** markvoelker has joined #openstack-infra | 15:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Include "delegate_to: localhost" in zuul_return examples https://review.openstack.org/630700 | 15:20 |
| *** markvoelker has quit IRC | 15:20 | |
| *** derekh has joined #openstack-infra | 15:21 | |
| *** efried1 has joined #openstack-infra | 15:24 | |
| *** apetrich has quit IRC | 15:25 | |
| *** efried has quit IRC | 15:25 | |
| *** efried1 is now known as efried | 15:25 | |
| corvus | mordred: the gerrit image build job failed the last two runs due to two different build errors. i find that curious | 15:29 |
| corvus | i've rechecked to see what #3 gets us | 15:29 |
| *** ekultails has quit IRC | 15:30 | |
| *** roman_g has quit IRC | 15:30 | |
| *** apetrich has joined #openstack-infra | 15:39 | |
| *** e0ne_ has quit IRC | 15:41 | |
| *** ekultails has joined #openstack-infra | 15:46 | |
| *** yboaron has quit IRC | 15:49 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 15:50 |
| *** e0ne has joined #openstack-infra | 15:51 | |
| *** apetrich has quit IRC | 15:54 | |
| *** weshay has joined #openstack-infra | 15:56 | |
| *** lpetrut has quit IRC | 15:58 | |
| *** kjackal has quit IRC | 15:59 | |
| *** armax has joined #openstack-infra | 16:02 | |
| *** eharney has joined #openstack-infra | 16:02 | |
| *** gfidente has quit IRC | 16:04 | |
| openstackgerrit | Merged openstack-infra/system-config master: Put opendev website content onto gitea homepage https://review.openstack.org/629917 | 16:05 |
| *** gfidente has joined #openstack-infra | 16:07 | |
| *** apetrich has joined #openstack-infra | 16:11 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add governance document https://review.openstack.org/622439 | 16:12 |
| mordred | corvus: ++ | 16:18 |
| *** pcaruana has quit IRC | 16:20 | |
| *** ykarel|away has quit IRC | 16:20 | |
| *** markmcd has joined #openstack-infra | 16:23 | |
| *** fuentess has joined #openstack-infra | 16:23 | |
| *** kjackal has joined #openstack-infra | 16:25 | |
| *** eharney has quit IRC | 16:25 | |
| *** sreejithp has quit IRC | 16:30 | |
| *** chandankumar is now known as codemonster | 16:31 | |
| *** sreejithp has joined #openstack-infra | 16:33 | |
| *** kopecmartin is now known as kopecmartin|off | 16:34 | |
| clarkb | corvus: I expect "ah ah ah" if it fails a third way :) | 16:37 |
| *** lpetrut has joined #openstack-infra | 16:37 | |
| clarkb | dpawlik isn't here anymore but we dont have a list of cpu flags | 16:37 |
| *** rfolco is now known as rfolco|brb | 16:37 | |
| clarkb | we've mostly relied on the clouds for that | 16:38 |
| *** corvus is now known as thecount | 16:38 | |
| thecount | mordred, clarkb: three failures! ah! ah! ah! http://logs.openstack.org/68/630468/9/check/system-config-build-image-gerrit/f9428c5/job-output.txt.gz#_2019-01-14_15_49_06_084400 | 16:38 |
| *** thecount is now known as corvus | 16:38 | |
| mordred | uhm | 16:39 |
| mordred | corvus: like, seriously? | 16:40 |
| *** eharney has joined #openstack-infra | 16:40 | |
| *** zxiiro is now known as zxiiro-away | 16:40 | |
| corvus | mordred: yeah... the first was "javascript is hard to build" the second was "i need more memory" the third was "i can't checkout the stable-2.15 git branch" | 16:41 |
| *** gfidente has quit IRC | 16:41 | |
| mordred | corvus: ara gives slighly more words - but they're still not helpful | 16:41 |
| corvus | that seems like a trend line -- and entropy seems to be *increasing* | 16:41 |
| mordred | http://logs.openstack.org/68/630468/9/check/system-config-build-image-gerrit/f9428c5/ara-report/result/4f00e296-8f74-4340-9ae9-e9bc8a9bc646/ | 16:42 |
| mordred | corvus: to be fair, that's what the second law of thermodynamics tells us | 16:42 |
| *** hrw has joined #openstack-infra | 16:48 | |
| hrw | fungi: thanks | 16:48 |
| *** yamamoto has joined #openstack-infra | 16:49 | |
| *** codemonster is now known as chkumar|out | 16:49 | |
| fungi | hrw: my pleasure. i actually try to keep on top of putting actual wiki users into the autopatrol group so that i can more easily spot spammers in the unpatrolled edits list | 16:50 |
| hrw | ) | 16:50 |
| hrw | :) | 16:50 |
| *** kjackal has quit IRC | 16:51 | |
| *** roman_g has joined #openstack-infra | 16:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 16:54 |
| *** yamamoto has quit IRC | 16:54 | |
| *** rpittau has quit IRC | 16:57 | |
| *** ykarel|away has joined #openstack-infra | 17:00 | |
| clarkb | http://paste.openstack.org/show/742333/ is the trimmed and made more readable puppet logs from review-dev01 on friday | 17:01 |
| clarkb | fungi: corvus mordred ^ | 17:01 |
| *** eernst has joined #openstack-infra | 17:01 | |
| clarkb | one issue is that we failed to rename some files on the init step? specifically /home/gerrit2/review_site/lib/.bcpkix-1.52.jar.backup and/or bcpkix-1.52.jar and then the other is that we ran the reindex | 17:02 |
| *** ginopc has quit IRC | 17:02 | |
| fungi | i wonder if the exec there expects something/someone to clean up .bcpkix-1.52.jar.backup afterward | 17:03 |
| clarkb | and ya the puppet seems to assume any new war will require a full offline reindex | 17:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 17:03 |
| *** gfidente has joined #openstack-infra | 17:05 | |
| *** ykarel|away has quit IRC | 17:05 | |
| *** eernst has quit IRC | 17:06 | |
| clarkb | fungi: I think the war notices we've add the files ourselves so it moves them aside (the .backup files) then grabs the copies it wants (then our puppet undoes that later?) | 17:06 |
| clarkb | fungi: my guess is that our puppet sets perms on the file such taht gerrit couldn't move the file/rename it | 17:06 |
| fungi | sounds likely | 17:06 |
| *** hrw has left #openstack-infra | 17:06 | |
| clarkb | -rw-r--r-- 1 root root 622849 Mar 9 2015 bcpkix-1.52.jar | 17:07 |
| *** eernst has joined #openstack-infra | 17:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 17:09 |
| fungi | puppet is running as root, right? maybe .bcpkix-1.52.jar.backup isn't writable? | 17:10 |
| clarkb | yes puppet runs as root | 17:10 |
| *** pgaxatte has quit IRC | 17:11 | |
| clarkb | -rw-r--r-- 1 gerrit2 gerrit2 622849 Mar 9 2015 .bcpkix-1.52.jar.backup or maybe it is mad that it already exists? | 17:11 |
| *** eernst has quit IRC | 17:12 | |
| clarkb | actually no I think it must be because it is trying to mv the source file (eg rename it) | 17:12 |
| clarkb | and that requires root, but if the gerrit init runs as gerrit2 then it will fail | 17:12 |
| * clarkb doubel checks the user on that exec | 17:13 | |
| clarkb | ya that exec runs as gerrit2 | 17:13 |
| *** eernst has joined #openstack-infra | 17:13 | |
| clarkb | let me push up a fix for the file ownership thing | 17:13 |
| *** e0ne has quit IRC | 17:14 | |
| *** lpetrut has quit IRC | 17:14 | |
| *** pcaruana has joined #openstack-infra | 17:15 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit master: Set java lib user ownership to gerrit2 https://review.openstack.org/630739 | 17:16 |
| clarkb | fungi: ^ that won't fix it for the existing files on disk. We need to chown those, but subsequent runs should be done properly I think | 17:16 |
| clarkb | still doesn't address the reindex. I'm thinking for that maybe we just remove the block of code from the module entirely and if we have to do a manual reindex we are not using puppet anyways since it takes all day? | 17:17 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add upload-puppetforge role https://review.openstack.org/627553 | 17:17 |
| clarkb | tobias-urdin: ^ fyi | 17:17 |
| *** eernst has quit IRC | 17:18 | |
| *** eernst has joined #openstack-infra | 17:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 17:20 |
| clarkb | hrm but then you can't do a fresh install | 17:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add gerrit.googlesource.com to zuul using git driver https://review.openstack.org/630740 | 17:20 |
| mordred | corvus: ^^ there's a patch to add gerrit upstream as a git driver source | 17:20 |
| clarkb | I98ce9ff87570a86e9b58b38d0605313ee5ee2758 added the code that we are having trouble with | 17:20 |
| mordred | corvus: (at elast I think that's what it does) | 17:20 |
| tobias-urdin | clarkb: thanks! should we merge https://review.openstack.org/#/c/627573/ now or wait until we have a release ready that we can test it with right away? | 17:20 |
| clarkb | I think the issue here is we can't distinguish between first ever gerrit init init and first gerrit initial init on a new war | 17:20 |
| clarkb | one option is to set offline_reindex to true when bringing up a new server, then switch it to false once it is installed | 17:21 |
| clarkb | I'll put a revert of that change up with a docs note that ^ is needed to bootstrap new install | 17:21 |
| mordred | clarkb: ++ | 17:23 |
| corvus | clarkb, fungi: can you +3 https://review.openstack.org/630467 ? then i'll start porting that over to zuul-jobs | 17:23 |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| *** eernst has quit IRC | 17:24 | |
| corvus | mordred: +2 (and gee whiz we need to refactor those variables, huh?) | 17:24 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit master: Revert "Create index on fresh install" https://review.openstack.org/630742 | 17:24 |
| clarkb | infra-root ^ that change and 630739 are hopefully enough of a bandaid to make limping along on puppet + gerrit 2.13 possible | 17:24 |
| mordred | corvus: yeah. I think that refactor will be called "ansible" :) | 17:24 |
| clarkb | corvus: looking | 17:24 |
| *** jpich has quit IRC | 17:25 | |
| clarkb | tobias-urdin: I think we should get that in then hopefully you'll be able to do a relase soonish | 17:25 |
| *** eernst has joined #openstack-infra | 17:25 | |
| tobias-urdin | clarkb: ack | 17:26 |
| clarkb | mordred: corvus: we need that so that relative paths in the dockerfile are relative to the gerrit repo? | 17:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add repos from gerrit.googlesource.com to zuul https://review.openstack.org/630744 | 17:27 |
| *** yamamoto has joined #openstack-infra | 17:27 | |
| mordred | clarkb: it's more that we want the gerrit repo to be the directory that gets sent to the docker daemon so that dockerfile commands like "COPY" are copying files from it | 17:28 |
| mordred | clarkb: but we want the dockerfile itself to not be in the gerrit repo | 17:28 |
| clarkb | ya so relative path for eg COPY | 17:28 |
| mordred | yah | 17:29 |
| *** eernst has quit IRC | 17:30 | |
| *** eernst has joined #openstack-infra | 17:32 | |
| *** yamamoto has quit IRC | 17:33 | |
| *** lpetrut has joined #openstack-infra | 17:33 | |
| *** ccamacho has quit IRC | 17:36 | |
| *** eernst has quit IRC | 17:36 | |
| *** eernst has joined #openstack-infra | 17:38 | |
| ssbarnea|rover | interesting subject https://github.com/yaml/pyyaml/issues/193#issuecomment-452941973 - should we try to lobby a little bit to steer the ship into the right direction? | 17:38 |
| *** yamamoto has joined #openstack-infra | 17:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 17:39 |
| *** eernst has quit IRC | 17:42 | |
| prometheanfire | mordred: when is the next release for openstacksdk set to be cut (asking for dogpile.cache uncapping) https://storyboard.openstack.org/#!/story/2004605 | 17:43 |
| mordred | prometheanfire: was gonna send in a release request today | 17:43 |
| *** eernst has joined #openstack-infra | 17:44 | |
| prometheanfire | cool | 17:44 |
| clarkb | ssbarnea|rover: ingy is the creator of yaml fwiw. Basically yaml was intended fro the unsafe use case | 17:45 |
| clarkb | ssbarnea|rover: just because most of us use yaml in a safe by default capactiy today doesn't mean the language isn't explicitly designed to have that functionality | 17:45 |
| clarkb | (from a technically correct standpoint ingy is right. From a lets be pragmatic standpoint with everyone using yaml today maybe we should be safe by default in the libraries that read yaml) | 17:45 |
| *** eernst has quit IRC | 17:48 | |
| *** eernst has joined #openstack-infra | 17:50 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 17:52 |
| openstackgerrit | Merged openstack-infra/system-config master: Add support for passing path to docker build https://review.openstack.org/630467 | 17:52 |
| fungi | yes, it's like saying python is unsafe because someone might use the pickle module from its stdlib on untrusted data, and so unpickle() should be made "safe" by default | 17:54 |
| *** eernst has quit IRC | 17:55 | |
| *** jamesmcarthur has joined #openstack-infra | 17:55 | |
| corvus | i'm certainly not going to pile on to that issue, especially since i agree with ingy | 17:56 |
| *** eernst has joined #openstack-infra | 17:56 | |
| *** electrofelix has quit IRC | 17:57 | |
| *** eernst has quit IRC | 17:57 | |
| *** eernst has joined #openstack-infra | 17:58 | |
| *** gfidente has quit IRC | 17:58 | |
| fungi | same | 17:59 |
| *** yamamoto has quit IRC | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| fungi | in my opinion, it's not serialization libraries like this which are "unsafe" but rather the dangerous practice (which similarly infected the php ecosystem for many, many years) of people copy-pasting code examples they find on the internet without reading the documentation for and understanding the libraries they use | 18:01 |
| *** jamesmcarthur has quit IRC | 18:01 | |
| ssbarnea|rover | clearly is debatable subject, I guess that at some point someone with add a linting check for use of pyyaml loader without mentioning clear safe/unsafe. | 18:02 |
| *** gfidente has joined #openstack-infra | 18:02 | |
| clarkb | heh my puppet-gerrit change failed rspec testing because gerrit index didn't happen | 18:02 |
| * clarkb fixes | 18:02 | |
| *** gfidente is now known as gfidente|afk | 18:02 | |
| *** eernst has quit IRC | 18:03 | |
| fungi | ssbarnea|rover: github has already e-mailed every pyyaml-using project hosted there warning them that they have a "potential security vulnerability" in their software | 18:03 |
| *** lpetrut has quit IRC | 18:03 | |
| fungi | that's what seems to have renewed interest in that particular issue in recent weeks | 18:04 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit master: Revert "Create index on fresh install" https://review.openstack.org/630742 | 18:06 |
| clarkb | I think ^ will pass tests now | 18:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 18:06 |
| *** jamesmcarthur has joined #openstack-infra | 18:06 | |
| *** eernst has joined #openstack-infra | 18:09 | |
| *** eernst has quit IRC | 18:11 | |
| *** panda is now known as panda|off | 18:12 | |
| *** jpena is now known as jpena|off | 18:12 | |
| *** rfolco|brb is now known as rfolco | 18:14 | |
| *** yamamoto has joined #openstack-infra | 18:15 | |
| *** e0ne has joined #openstack-infra | 18:16 | |
| mordred | fungi: heaven help anyone who is using yaml.load on purpose :) | 18:19 |
| fungi | yeah, like, you know, on trusted serialized data | 18:19 |
| fungi | mordred: github is warning any project which declares a dependency on pyyaml, regardless of whether they actually call load() | 18:20 |
| clarkb | so much email | 18:20 |
| corvus | mordred: yeah, that was sort of glyph's point, which, he figures anyone who has poked their eyes out with the zodb should understand. seems he was right :) | 18:20 |
| mordred | yeah. reminds me of that thing from a couple of years ago where "use of md5" started to trigger people's security linters | 18:21 |
| mordred | or maybe it was use of sha1 - or whatever it was | 18:21 |
| clarkb | md5 came back up again recently on the openstack list | 18:21 |
| *** kjackal has joined #openstack-infra | 18:21 | |
| mordred | god really? | 18:21 |
| clarkb | (for the same reason, some security check say md5 and went bad bad software) | 18:21 |
| clarkb | ya | 18:21 |
| *** smarcet has joined #openstack-infra | 18:21 | |
| fungi | right. people still not able to wrap their heads around collision resistance vs first and second preimage resistance | 18:22 |
| smarcet | fungi: mordred: clarkb: afternoon please review https://review.openstack.org/#/c/595839/ | 18:22 |
| *** hwoarang has quit IRC | 18:22 | |
| *** wolverineav has joined #openstack-infra | 18:22 | |
| mordred | corvus: oh good. this gerrit build error is one of the ones we've seen before | 18:23 |
| corvus | hey it's the javascript one! | 18:24 |
| corvus | ok, so we have built the gerrit war "normally" right? | 18:24 |
| *** eernst has joined #openstack-infra | 18:25 | |
| *** hwoarang has joined #openstack-infra | 18:25 | |
| mordred | although the javascript thing is just complaining about a missing source map - the error error seems to be something something WorkResponse? | 18:25 |
| mordred | corvus: I've built it with this dockerfile | 18:25 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Migrate OpenStackID dev server to php7 https://review.openstack.org/625640 | 18:25 |
| corvus | mordred: i was reading that as "the build thingy didn't say 'i built' instead it spewed error output" | 18:25 |
| *** eernst has quit IRC | 18:26 | |
| mordred | yah - that's probably the right reading | 18:26 |
| *** eernst has joined #openstack-infra | 18:26 | |
| mordred | corvus: I'm doing a local build again ... if it works, then I think investigating git state differences (I set the git repo state up by hand and haven't done a pull or anything) | 18:27 |
| corvus | mordred: though perhaps still the js thing is a red herring... hard to say. i don't see anything *else* that looks like an error | 18:27 |
| mordred | yeah - me either | 18:27 |
| corvus | mordred: yah. you can run your "repos" playbook by hand too, right? | 18:27 |
| mordred | yah | 18:28 |
| ssbarnea|rover | fungi: regarding deprecation obsolete workflows. i wonder if we cannot speed-up the obsolescence of these workflow. let me know how (stupid) you find my idea... | 18:28 |
| ssbarnea|rover | fungi: add big red warning message at top of deprecated roles with 10 min delay. every other week we add other 10 min to the delay. | 18:30 |
| fungi | stupid isn't the word i would use, but it does tie up ci resources with an otherwise unnecessary delay (as well as with job timeouts which then end up getting rechecked over and over) | 18:31 |
| fungi | we ought to be able to identify frequent users via logstash queries | 18:32 |
| ssbarnea|rover | i am saying this because in my experience most people delay any refactoring until they are forced to do it, regardless how much effort you put in trying to convince. | 18:32 |
| ssbarnea|rover | fungi: yep, the recheck is an issue. | 18:32 |
| fungi | they likely won't see an artifically-induced delay as forcing them to do something, but rather as hostility | 18:33 |
| *** ekultails has quit IRC | 18:33 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Migrate OpenStackID dev server to php7 https://review.openstack.org/625640 | 18:35 |
| ssbarnea|rover | fungi: i was trying be creative, not hostile :D maybe someone else has a better idea. | 18:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Add puppetforge publish and add to puppet release https://review.openstack.org/627573 | 18:36 |
| ssbarnea|rover | this issue applies to any breaking change that needs adoption, without a clear deadlines a full migration is unlikely to ever reach 100%/ | 18:36 |
| fungi | sure, i don't think you were trying to be hostile at all, just saying that's how it will likely be perceived by developers who, after spending a while digging into why their jobs are taking so long, discover it's because a sleep was added specifically to annoy them into doing something different | 18:37 |
| dmsimard | Odds are they might not even notice the delay | 18:37 |
| dmsimard | The standard deviation from running in different clouds etc | 18:37 |
| fungi | they'll notice it once their jobs start hitting timeouts and they recheck a dozen times | 18:38 |
| dmsimard | ianw had an interesting approach when there was the changes for readthedocs | 18:38 |
| dmsimard | He created a new job like "your-rtd-job-requires-attention" | 18:38 |
| ssbarnea|rover | dmsimard: haha, true. Maybe altering gerrit msg? replacing the green SUCCESS label, with something orange? | 18:38 |
| *** ekultails has joined #openstack-infra | 18:38 | |
| ssbarnea|rover | dmsimard: that job was not really a bad idea, it made me fix few things. | 18:39 |
| fungi | ssbarnea|rover: the traditional solution is to print a deprecation warning, send some announcements about end of life for maintaining it, maybe try to put together a list of frequent users (if it's not too much work), and then remove it at the deadline | 18:39 |
| *** yamamoto has quit IRC | 18:39 | |
| fungi | people whose jobs break at that point are likely to notice, and have less to complain about than if we're just adding a sleep to make jobs annoying | 18:40 |
| fungi | i personally don't want to run a ci system which sees annoying users as a solution to its problems | 18:40 |
| ssbarnea|rover | is it possible to add this your-job-definition-is-deprecarted job? | 18:41 |
| AJaeger | ssbarnea|rover: we did this with the rfc change - and have very little changes ;( | 18:41 |
| *** smarcet has quit IRC | 18:42 | |
| AJaeger | ssbarnea|rover: check template docs-on-readthedocs-failing | 18:42 |
| fungi | ssbarnea|rover: communicating deprecation warnings to people who are in a position to act on them is an interesting challenge, so i think it's a good topic of discussion | 18:42 |
| AJaeger | ssbarnea|rover: argh, should have read dmsimard comment earlier, sorry for duplicate. | 18:43 |
| *** yamamoto has joined #openstack-infra | 18:43 | |
| fungi | we can likely use some introspection in zuul to identify loaded configuration which uses roles/playbooks flagged for deprecation | 18:44 |
| fungi | that might be a useful feature | 18:44 |
| clarkb | could just add a warning to the roles the emit a deprecated message to the console logs, then generate ar eport of all the jobs that have deprecated messages from elasticsearch | 18:44 |
| fungi | as for identifying references to deprecated tools and repositories, that probably depends on source code searching | 18:44 |
| dmsimard | The part about getting metrics regarding who uses what roles is interesting but challenging outside openstack | 18:45 |
| fungi | clarkb: yeah, that gets 1. anyone who looks at the logs and can act on the deprecation notice (or notify someone who in turn can), and projects with logs indexed in logstash which run those jobs with any regularity, as long as we know who to reach out to for them | 18:46 |
| ssbarnea|rover | clarkb: i like the deprecated message approach because we can mix it with an e-r query and get a clear idea about the trend. | 18:46 |
| fungi | do we index console logs for all jobs across the board? | 18:46 |
| ssbarnea|rover | from my experience most people are ignoring most of warnigns in logs. | 18:47 |
| dmsimard | ssbarnea|rover: oh, using logstash to track deprecation warnings is clever | 18:47 |
| ssbarnea|rover | fungi: afaik, console log is indexed by all jobs. | 18:47 |
| clarkb | fungi: yes | 18:47 |
| *** remi_ness has joined #openstack-infra | 18:47 | |
| fungi | cool, then as long as they run those jobs with some regularity we can probably find them, just need to know who to reach out to about them | 18:47 |
| fungi | and as dmsimard points out, we're of course blind to anyone consuming those outside our zuul deployment | 18:48 |
| fungi | so still need to couple with announcements | 18:48 |
| clarkb | https://review.openstack.org/#/c/630742/ and https://review.openstack.org/#/c/630739/ should fix (I hope) the puppet-gerrit stuff we found on friday and both are zuul +1 | 18:48 |
| clarkb | also there is a stack of gerrit changes at https://review.openstack.org/#/c/630390/ to get some wars built | 18:49 |
| clarkb | if we can get both sets of changes merged I'll update system-config to use the actual war on review-dev, make sure it all works then we can look at updating prod | 18:49 |
| clarkb | fungi: ya | 18:49 |
| fungi | as an up-side, i think when we do remove any deprecated roles/playbooks, zuul will notice via configuration errors (either commenting on the removal or listing the config errors in the dashboard)? | 18:50 |
| mordred | corvus: I have "good news" | 18:51 |
| clarkb | fungi: I believe that is correct | 18:51 |
| fungi | obviously that doesn't help for removals of tools like the zuul-cloner executable | 18:51 |
| mordred | corvus: the local build I did worked | 18:51 |
| mordred | corvus: it also exhibited the could-not-find-source-map warning, but was otherwise fine with life | 18:51 |
| corvus | oh, hrm, so that was a red herring? | 18:52 |
| corvus | mordred: i guess that means something is wrong in docker on a zuul vm? | 18:52 |
| mordred | maybe? but now we get to figure out why it's working for me locally and not on zuul vms | 18:52 |
| clarkb | fungi: openid login on the wiki is broken because login.launchpad.net has a broken ssl cert | 18:53 |
| mordred | corvus: yah. lemme go look at git repo state and stuff to | 18:53 |
| clarkb | fungi: I'm guessing we should take this as an indication to switch to ubuntu one and update the openid db tables as necessary? | 18:53 |
| fungi | clarkb: i expect so, yes | 18:53 |
| clarkb | re ^ the infra meeting agenda email might be late this week :) | 18:54 |
| fungi | i'll see if i can do that real quick | 18:54 |
| clarkb | fungi: thanks | 18:54 |
| *** remi_ness has quit IRC | 18:54 | |
| mordred | corvus: there don't seem to be any changes git-repo state-wise - so nothing upstream has shifted | 18:55 |
| clarkb | mordred: different base images somehow? | 18:56 |
| clarkb | (like maybe you pulled a debian or whatever from one location and that is different than dockerhub?) | 18:56 |
| corvus | mordred, clarkb, fungi: okay, so i just found this thing which seem to contradict dockerhub documentation by saying that you actually can delete a tag: https://devopsheaven.com/docker/dockerhub/2018/04/09/delete-docker-image-tag-dockerhub.html | 18:57 |
| corvus | and i verified it works | 18:57 |
| clarkb | huh | 18:58 |
| clarkb | I wonder if they intend to remove that functionality at some point hence the mismatch in docs? or added it and didn't update docs to match | 18:58 |
| corvus | so should i re-attempt to delete obsolete gate pipeline tags, or should we skip supporting that since it's apparently not documented | 18:58 |
| clarkb | in either case, maybe we use that until it breaks? | 18:58 |
| clarkb | corvus: my personal preference would be to avoid having hundreds/thousands/more tags over time if we can avoid it | 18:59 |
| corvus | okay, i'll work on that some more | 18:59 |
| corvus | i'll delay moving this into zuul-jobs roles until we work it out | 19:00 |
| corvus | (i discovered this *while* moving into zuul-jobs; i have no idea why i didn't find this last week) friday) | 19:00 |
| *** diablo_rojo has joined #openstack-infra | 19:01 | |
| fungi | 11343 rows to update | 19:02 |
| mordred | clarkb: that's a good thought (base image diff) - I'm pushing my local openjdk:8 up to dockerhub as emonty/openjdk-8 - and will submit a new version of the patch that uses that instead of openjdk:8 | 19:02 |
| fungi | backing up mediawiki trove db via mysqldump now | 19:03 |
| *** wolverineav has quit IRC | 19:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 19:05 |
| clarkb | fungi: let me know if I can help. Though I promised kids early lunch so will have to step out soon | 19:05 |
| fungi | updated $wgOpenIDForcedProvider to https://login.ubuntu.com/+openid in /srv/mediawiki/Settings.php | 19:05 |
| fungi | generating update query set via: | 19:06 |
| fungi | sudo -H mysql -e 'select uoi_openid from user_openid where uoi_openid like "%login.launchpad.net%"' openstack_wiki|sed 's/.*/update users set openid="\0" where openid="\0";/'|sed 's/launchpad.net/ubuntu.com/'>openid_update.sql | 19:06 |
| fungi | sorry, was user_openid not users in the update query | 19:07 |
| *** kjackal has quit IRC | 19:07 | |
| fungi | and uoi_openid for the set | 19:07 |
| fungi | as well as in the where | 19:08 |
| *** pcaruana has quit IRC | 19:08 | |
| fungi | all updated | 19:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add release note for zuul_return action plugin update https://review.openstack.org/630760 | 19:09 |
| corvus | fungi: i just logged in | 19:09 |
| fungi | restarting apache2 service now so that the mediawiki settings is reloaded | 19:09 |
| fungi | confirmed, i've logged in successfully too | 19:10 |
| fungi | clarkb: ^ you should be all set now | 19:10 |
| clarkb | fungi: yup I'm logged int too | 19:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 19:11 |
| clarkb | will edit a page to make sure my user didn't change (but it reported what I expected so don't think that happened) | 19:11 |
| corvus | fungi: i think the mediawiki settings took immediate effect | 19:11 |
| fungi | corvus: good to know. sometimes they don't due to mod_php caching the settings | 19:12 |
| clarkb | I am able to edit a page as expected too | 19:12 |
| corvus | fungi: between your two sql statements, the wiki was using ubuntu one, but not recognizing my account. | 19:12 |
| fungi | noted | 19:12 |
| fungi | i did my best to minimize that race as much as possible anyway, just in case it did | 19:12 |
| corvus | i canceled the process to avoid creating a second account; though there's a small possibility someone else didn't. unclear what would happen in that case. :) | 19:12 |
| fungi | and yeah, my wiki admin perms seem to have been preserved | 19:12 |
| corvus | fungi: i figured, which is why i didn't bother to mention it at the time. :) | 19:13 |
| *** yamamoto has quit IRC | 19:13 | |
| fungi | corvus: i think they'll end up with duplicate accounts and then... dunno. error? random? | 19:13 |
| openstackgerrit | Merged openstack-infra/zuul master: sql: add buildset uuid column https://review.openstack.org/630034 | 19:13 |
| fungi | i suppose i could work out the sql for finding rows with more than one of the same value for the uoi_openid column | 19:14 |
| *** yamamoto has joined #openstack-infra | 19:14 | |
| mordred | clarkb: sigh. there is actually no difference between my local openjdk:8 and what's on dockerhub :( | 19:15 |
| mordred | and I'm also using upstream docker just like the job is | 19:18 |
| *** e0ne has quit IRC | 19:19 | |
| mordred | I'm going to put in a hold | 19:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki master: Update default OpenID provider to ubuntu.com https://review.openstack.org/630761 | 19:24 |
| fungi | #status log updated openid provider for wiki.openstack.org from login.launchpad.net to login.ubuntu.com | 19:24 |
| openstackstatus | fungi: finished logging | 19:24 |
| *** remi_ness has joined #openstack-infra | 19:25 | |
| *** yamamoto has quit IRC | 19:27 | |
| *** wolverineav has joined #openstack-infra | 19:29 | |
| *** wolverineav has quit IRC | 19:32 | |
| *** wolverineav has joined #openstack-infra | 19:32 | |
| *** yamamoto has joined #openstack-infra | 19:33 | |
| *** yamamoto has quit IRC | 19:35 | |
| *** yamamoto has joined #openstack-infra | 19:37 | |
| *** signed8bit has joined #openstack-infra | 19:38 | |
| *** yamamoto has quit IRC | 19:45 | |
| *** yamamoto has joined #openstack-infra | 19:49 | |
| *** e0ne has joined #openstack-infra | 20:01 | |
| *** yamamoto has quit IRC | 20:04 | |
| *** remi_ness has quit IRC | 20:05 | |
| *** yamamoto has joined #openstack-infra | 20:08 | |
| *** yamamoto has quit IRC | 20:11 | |
| *** yamamoto has joined #openstack-infra | 20:12 | |
| corvus | clarkb: i just realized an issue with the tag cleanup. i was going to delete all tags older than the tag we're promoting, however, there could be multiple branches at play. if the stable/foo branch creates 'foo' docker tags, then a change on master can't go deleting all the older tags because it might delete a 'foo' tag which is about to be promoted. | 20:18 |
| smcginnis | I've been seeing a lot of gate failures with SSH time outs and errors assigning floating IPs. Anyone aware of any current issues that may be related? | 20:18 |
| clarkb | corvus: oh hrm | 20:19 |
| clarkb | smcginnis: I believe that is a known neutron bug | 20:19 |
| smcginnis | OK, thanks clarkb. I'll see if I can track that down. | 20:19 |
| corvus | clarkb: i could make the tags "change_<branch>_<number>" maybe? | 20:20 |
| corvus | then delete all "change_<branch>_*" tags older than ours? | 20:20 |
| clarkb | smcginnis: https://bugs.launchpad.net/neutron/+bug/1811515 that one I think | 20:20 |
| openstack | Launchpad bug 1811515 in neutron "SSH to FIP fails in CI jobs" [Critical,Confirmed] | 20:20 |
| corvus | (i bet we'd have to transpose '/' chars in the 'branch' part of that) | 20:20 |
| clarkb | corvus: I think that is safe with supercedenet pipeline | 20:20 |
| smcginnis | Ah, looks like it could be it. Thanks for that. | 20:21 |
| corvus | clarkb: yeah. that would take care of branch cases. if, however, you had a more esoteric case where you built more than one tag from the same branch in different jobs, i think we'd lose. like if you built "myapp:ubuntu" and "myapp:redhat".... | 20:22 |
| corvus | i guess that's not so esoteric. | 20:22 |
| corvus | ok, here are things we can do easily: 1) delete the change_tag of thing we just promoted. 2) delete change_tags older than a certain period (like 24 hours). | 20:25 |
| corvus | since this is built for near-immediate promotion after gate, i think #2 is quite safe | 20:25 |
| corvus | so with the combination of those two items, we should be able to keep change_tag pollution to a mininum. | 20:25 |
| clarkb | ya 2 should be safe | 20:30 |
| *** bgmccollum has joined #openstack-infra | 20:31 | |
| sshnaidm | pabelanger, would be nice to have some explanation for -1 :) https://review.openstack.org/#/c/630649/ | 20:36 |
| *** whoami-rajat has quit IRC | 20:40 | |
| clarkb | fungi: have the link to our docs on how to format third party ci results for table inclusion handy? | 20:43 |
| clarkb | I'm not finding it after a quick search | 20:43 |
| *** smarcet has joined #openstack-infra | 20:44 | |
| fungi | clarkb: last example in the https://docs.openstack.org/infra/system-config/third_party.html#the-jenkins-gerrit-trigger-plugin-way section | 20:45 |
| fungi | "To format the result’s message in a way that works with the current OpenStack Gerrit GUI parser..." | 20:45 |
| *** kjackal has joined #openstack-infra | 20:45 | |
| clarkb | thanks | 20:45 |
| *** eernst has quit IRC | 20:46 | |
| *** eernst has joined #openstack-infra | 20:49 | |
| *** eernst has quit IRC | 20:53 | |
| *** smarcet has quit IRC | 20:53 | |
| clarkb | I've approved the gerrit 2.13.12 stack to get war built | 20:55 |
| *** eernst has joined #openstack-infra | 20:55 | |
| *** smarcet has joined #openstack-infra | 20:55 | |
| *** eernst has quit IRC | 20:55 | |
| *** kjackal has quit IRC | 20:55 | |
| *** eernst has joined #openstack-infra | 20:55 | |
| clarkb | reviews on https://review.openstack.org/#/c/630739/ and https://review.openstack.org/#/c/630742/ much appreciated to get the war deployed | 20:55 |
| clarkb | ianw ^ when you wake that second one may interest yuo (its reverting a change you had pushed) | 20:56 |
| clarkb | (I think ultimately two thigns are at odds there, bootstrapping new server and upgrading existing server. We can hack that with the offline_reindex flag as documented in that change and applied via the rspec tests) | 20:56 |
| *** yamamoto has quit IRC | 20:56 | |
| *** yamamoto has joined #openstack-infra | 20:57 | |
| fungi | clarkb: i _think_ (though could be wrong) that modern gerrit releases already detect whether a reindex is required. i wonder if we could trigger off that somehow? | 20:59 |
| clarkb | fungi: I think most of them are happy with an online reindex instead of offline | 20:59 |
| clarkb | not sure if that is a detection thing or just a rule with testing? | 21:00 |
| clarkb | in this case we need to distinguish between first ever install and an upgrade | 21:00 |
| clarkb | and I think setting a flag to enable that is probably good enough? | 21:00 |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** smarcet has quit IRC | 21:00 | |
| *** yamamoto has quit IRC | 21:03 | |
| *** hamzy_ has joined #openstack-infra | 21:03 | |
| *** smarcet has joined #openstack-infra | 21:03 | |
| *** smarcet has quit IRC | 21:03 | |
| *** sparkycollier_ has joined #openstack-infra | 21:04 | |
| *** Dobroslaw_ has joined #openstack-infra | 21:04 | |
| fungi | well, when i looked at the error log for the aborted upgrade, it complained that it needed an offline reindex run... but that might be because there was already an offline index which was partly performed and then aborted when the puppet exec timeout was reached? | 21:04 |
| *** JpMaxMan_ has joined #openstack-infra | 21:04 | |
| *** mwhahaha_ has joined #openstack-infra | 21:04 | |
| fungi | might be gerrit can't perform an online reindex to recover from an aborted offline reindex | 21:04 |
| *** smarcet has joined #openstack-infra | 21:05 | |
| clarkb | yes, it may move the old index aside for manual recovery but I don't think it can recover automatically if the existing index is corrupted | 21:06 |
| clarkb | aiui upgrades within the same release of gerrit are never supposed to require offline reindexing | 21:06 |
| clarkb | we can also test that by downgrading, doing a full reindex (that returns us to old situation) then upgrade again with these changes in place | 21:06 |
| *** HenryG_ has joined #openstack-infra | 21:07 | |
| mordred | clarkb: I think that's a good idea | 21:08 |
| clarkb | in the past upgrades between the "major" releases required offlien reindexing then in 2.12 or 2.13 the plan forward was to not require that. Except then 2.14 added the notedb transition but they've since updated that upgrade to allow 2.13-> 2.15 direct without any offline reindex in between aiui | 21:09 |
| * mordred is so looking forward to that | 21:11 | |
| *** Dobroslaw has quit IRC | 21:11 | |
| *** sparkycollier has quit IRC | 21:11 | |
| *** mgoddard has quit IRC | 21:11 | |
| *** JpMaxMan has quit IRC | 21:11 | |
| *** mwhahaha has quit IRC | 21:11 | |
| *** onovy has quit IRC | 21:11 | |
| *** HenryG has quit IRC | 21:11 | |
| *** hamzy has quit IRC | 21:11 | |
| *** jroll has quit IRC | 21:11 | |
| *** sparkycollier_ is now known as sparkycollier | 21:11 | |
| *** JpMaxMan_ is now known as JpMaxMan | 21:11 | |
| *** mwhahaha_ is now known as mwhahaha | 21:11 | |
| *** Dobroslaw_ is now known as Dobroslaw | 21:11 | |
| *** jroll has joined #openstack-infra | 21:12 | |
| clarkb | dmsimard: fwiw I think we had decided that this behavior was possibly due to a chaneg in ansible using a lot more memory | 21:13 |
| clarkb | starting in late november early december. That memory then trips the governor more often (and also leads to swap use) | 21:14 |
| clarkb | but I don't think anyone tracked it to a specific release of ansible | 21:14 |
| clarkb | and things mostly work even with the swap? | 21:14 |
| *** yamamoto has joined #openstack-infra | 21:14 | |
| clarkb | also complicated jobs use more memory (not surprising), in particular when I measured it last month grenade and tripleo jobs used more memory and swapped more often | 21:15 |
| *** mgoddard has joined #openstack-infra | 21:15 | |
| dmsimard | clarkb: the logs show a gap of 9 minutes (during which I probably observed the zuul web console loop) http://paste.openstack.org/raw/742352/ | 21:15 |
| dmsimard | It looks like we're making the web console available before we know which executor will take care of the build ? | 21:16 |
| clarkb | dmsimard: I'm not familar enough with the govnernor to know if that may be to blame for that too | 21:16 |
| clarkb | lik it grabs the next job then checks if it is safe to run it? | 21:16 |
| *** efried has quit IRC | 21:16 | |
| clarkb | dmsimard: yes, as soon as a nodeset is allocated iirc | 21:16 |
| clarkb | but those particulars would be #zuul specific :) | 21:16 |
| dmsimard | yeah | 21:16 |
| * dmsimard goes back | 21:16 | |
| *** efried has joined #openstack-infra | 21:17 | |
| *** yamamoto has quit IRC | 21:17 | |
| clarkb | 2867896KB swap used on ze01, 2709228KB of that is the executor process itself not ansible processes | 21:19 |
| clarkb | bulk of it is in zuul itself then | 21:19 |
| mordred | clarkb, corvus: I've got a node held - I cannot, so far, see any differences between it and my laptop | 21:20 |
| dmsimard | you mentioned something in Ansible might have changed ? do you know when that was ? | 21:20 |
| dmsimard | looking at ram usage in the last 30 days is not terribly conclusive http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&panelId=28&fullscreen&from=now-30d&to=now | 21:20 |
| corvus | dmsimard: between september and november, iirc. | 21:20 |
| clarkb | dmsimard: end of novemeber/early december is when we see the shift in our graphs. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64005&rra_id=all | 21:20 |
| corvus | it's unclear when we restarted with various ansible versions, so is rathre hard to pin down. | 21:21 |
| clarkb | there is 3.5GB of memory available on that host so I think this is mostly zuul-executor having stale pages paged out? | 21:21 |
| clarkb | dmsimard: though now that I think of it that memory use in ansible itself could've been related to how we read console logs | 21:22 |
| clarkb | which was updated as part of our trying to limit the memory costs there | 21:22 |
| dmsimard | yeah... looking at top, you can see kswapd working | 21:23 |
| clarkb | another thought I had was that buffers for the console streams were maybe related? | 21:24 |
| clarkb | as you reduce the number of pages needed to service those requests they can get paged out, just waiting for the next big stream requests | 21:25 |
| mordred | clarkb, dmsimard: we're also still eating a high memory cost to write the json console log file | 21:26 |
| clarkb | because other than those erquests there isn't really any memory usage by that daemon iirc | 21:26 |
| clarkb | mordred: ya but thats in the subprocesses and only happens at the end of a job right? | 21:26 |
| mordred | one of the proprosed solutions is to shift to writing out a yaml file by appending all the time | 21:26 |
| dmsimard | mordred: yeah I was about to mention zuul-json | 21:26 |
| clarkb | or is that part of the zuul-executor? | 21:26 |
| dmsimard | loading and writing that file every time is probably costly | 21:26 |
| clarkb | (right now zuul-executor is using the memory) | 21:26 |
| mordred | clarkb: it's in teh ansible-playbook process | 21:26 |
| clarkb | ya so that doesn't explain zuul-exucutor being the bulk of the swap | 21:26 |
| mordred | so yeah - that's probably not it from a memory usage perspective | 21:26 |
| clarkb | re zuul json, could we hack it and write bytes we know will result in valid jsob? | 21:27 |
| mordred | the log streaming itself should be happening line at a time, so I wouldn't expect the executor to read the whole thing in to memory .. | 21:27 |
| clarkb | mordred: unless some job has a 1GB line | 21:27 |
| mordred | clarkb: indeed | 21:27 |
| clarkb | mordred: if you asked me 6 years ago I'd say thats silly | 21:27 |
| clarkb | but now I know better :P | 21:28 |
| mordred | clarkb: I'd rather just switch to yaml - it'll be more human readable, and no less machine readable | 21:28 |
| clarkb | mordred: I dunno there is a big debate on that on github now :P but ya thats a fair point | 21:28 |
| mordred | :) | 21:28 |
| *** xek has quit IRC | 21:28 | |
| mordred | clarkb: I tend to disagree with most debates on github :) | 21:29 |
| dmsimard | Out of curiosity, have we measured the impact of leaving debug enabled ? | 21:29 |
| *** smarcet has quit IRC | 21:29 | |
| clarkb | dmsimard: ansible debug? | 21:30 |
| clarkb | (I'm not sure what debug you mean) | 21:30 |
| dmsimard | looking at ze01, executor.log is at 48835 lines while executor-debug.log is at 1524572 | 21:30 |
| clarkb | oh the python logging? | 21:30 |
| dmsimard | that's a couple orders of magnitude | 21:30 |
| clarkb | python logging should be fine | 21:31 |
| *** smarcet has joined #openstack-infra | 21:31 | |
| clarkb | that goes to disk a message at a time and the messages are not particularly large | 21:31 |
| clarkb | (there is some memory cost but not much) | 21:31 |
| dmsimard | it's more about the amount of messages than the size but yeah | 21:31 |
| clarkb | if that is a source of problems it will be filling our disks not causing the deamon to swap | 21:32 |
| mordred | corvus: https://github.com/angular/devkit/commit/fab734b1e910bfeeb9faad0d054031aaaebcffee | 21:34 |
| *** smarcet has quit IRC | 21:34 | |
| mordred | corvus: "# Workaround https://github.com/bazelbuild/bazel/issues/3645 | 21:34 |
| mordred | # Bazel doesn't calculate the memory ceiling correctly when running under Docker. | 21:34 |
| mordred | " | 21:34 |
| mordred | corvus: I re-ran the build on the held node and it failed in the same way | 21:34 |
| *** smarcet has joined #openstack-infra | 21:34 | |
| mordred | corvus: I'm going to try putting in an rc file like that and seeing if it'll do the right thing | 21:34 |
| *** kgiusti has left #openstack-infra | 21:35 | |
| corvus | mordred: i'm trying to relate this to success in docker on your laptop | 21:35 |
| corvus | mordred: any ideas? | 21:35 |
| mordred | corvus: my laptop has more ram | 21:36 |
| mordred | so I think the build is consuming more ram than the 8G - and doing so badly because it's autodetecting memory wrongly - but on my laptop I have enough ram so the mis-detection isn't an issue? | 21:37 |
| ianw | clarkb: heh, 2 years, 5 months ago on that gerrit puppet change. i can barely remember what i ate for breakfast :) | 21:38 |
| *** smarcet has quit IRC | 21:38 | |
| mordred | corvus: although that particularly config is WAY small, the first number, 3072, is ram - but let's see if it works | 21:39 |
| mordred | corvus: got past gerrit-server/libserver-class.jar | 21:39 |
| *** e0ne has quit IRC | 21:40 | |
| clarkb | ianw: I only remember what I had for breakfast because it was at lunch :) | 21:41 |
| mordred | corvus: INFO: Build completed successfully, 2374 total actions | 21:43 |
| corvus | mordred: congratulations! you win at computers! | 21:44 |
| mordred | corvus: I'm going to work on making that a parameter we can pass in | 21:44 |
| mordred | corvus: the ENV command in Dockerfile allows you to set an env var that will be used if set in the calling context, right? | 21:45 |
| ianw | infra-root: could i get one more look at https://review.openstack.org/#/c/629681/ which reverts the centos-7-nm node, tripleo testing was successful with NetworkManager. no jobs use this node type, i'll manually clean up after merge | 21:45 |
| corvus | mordred: yes; and i think they can default? | 21:46 |
| *** eernst has quit IRC | 21:46 | |
| clarkb | looks like the console streaming is done in a separate daemon and that doesn't swap so heavily | 21:47 |
| mordred | corvus: yeah. so I'm thinking doing an ENV BAZEL_OPTS "" and putting ${BAZEL_OPTS} after the bazel command - then in our invocation of docker build set BAZEL_OPTS to "--local_resources=4096,2.0,1.0" | 21:48 |
| clarkb | so something in mergers, executor, governor, update thread or disk accountant | 21:48 |
| mordred | corvus: which might need an additional field in our images dict - to be a dict of env vars to set on the shell command | 21:48 |
| corvus | mordred: that sounds reasonable | 21:49 |
| clarkb | ianw: I think if you go to min-ready: 0 then actually get to 0 nodes of that type, then delete the provider images and diskimages it will clean up mostly for you | 21:49 |
| *** jcoufal has quit IRC | 21:49 | |
| clarkb | ianw: otherwise I think you may need to manually cleanup zk for things like min ready requests that may have gone stale? | 21:50 |
| *** eharney has quit IRC | 21:51 | |
| clarkb | ianw: I've +2'd anyway since I think it would be avaluable learning experience if this actually does work out fine :) | 21:51 |
| clarkb | and fixing it is straightforward (we revert) | 21:51 |
| dmsimard | clarkb: the "du" (from the disk accountant I suppose) shows up in the top processes for a while from time to time | 21:52 |
| clarkb | dmsimard: for memory or cpu? | 21:52 |
| clarkb | (but that should also be captured by the du process not the zuul process) | 21:52 |
| dmsimard | I'll have to tell you next time I see it, I'd guess that was cpu | 21:52 |
| dmsimard | also, yes, it's separate | 21:52 |
| *** eernst has joined #openstack-infra | 21:53 | |
| clarkb | fwiw the zuul mergers don't have this problem which likely rules out somethign related to the merger thread | 21:53 |
| *** fuentess has quit IRC | 21:54 | |
| dmsimard | yeah, the "du" is mostly cpu. | 21:55 |
| dmsimard | 0.2% mem | 21:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 21:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image environment variables https://review.openstack.org/630777 | 21:57 |
| corvus | clarkb, dmsimard: i triggered sigusr2 twice if you want to look in the debug logs for objgraph stats | 21:57 |
| corvus | on ze01 | 21:57 |
| *** eernst has quit IRC | 21:57 | |
| *** smarcet has joined #openstack-infra | 21:58 | |
| pabelanger | sshnaidm: Oh, sorry. I did enter some comments via UI, but guess they didn't post | 21:59 |
| pabelanger | sshnaidm: adding them now | 21:59 |
| *** eernst has joined #openstack-infra | 21:59 | |
| clarkb | corvus: I don't see 'Beginning debug handler' in the executor debug log on ze01 | 22:01 |
| clarkb | yappi is installed | 22:01 |
| clarkb | as is objgraph | 22:02 |
| *** yamamoto has joined #openstack-infra | 22:02 | |
| clarkb | did we regress that feature sonmehow? | 22:02 |
| dmsimard | clarkb: http://paste.openstack.org/show/742356/ | 22:02 |
| corvus | clarkb: it's there | 22:03 |
| corvus | 2019-01-14 21:30:29,001 DEBUG zuul.stack_dump: Beginning debug handler | 22:03 |
| clarkb | bah I had a typo in Beginning sorry | 22:03 |
| dmsimard | I grep'd for stack_dump :p | 22:03 |
| *** eernst_ has joined #openstack-infra | 22:04 | |
| *** eernst has quit IRC | 22:04 | |
| dmsimard | I'm not sure how to interpret that data or recognize if it is normal or not, though :( | 22:06 |
| *** yamamoto has quit IRC | 22:06 | |
| *** imacdonn has quit IRC | 22:06 | |
| *** eernst_ has quit IRC | 22:06 | |
| *** eernst has joined #openstack-infra | 22:06 | |
| *** imacdonn has joined #openstack-infra | 22:06 | |
| clarkb | dmsimard: its giving you counts of objects by type, then on the second run its a delta in that count | 22:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 22:07 |
| *** yamamoto has joined #openstack-infra | 22:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 22:07 |
| dmsimard | the numbers are identical in both columns, though ? | 22:07 |
| mordred | corvus: ^^ ENV is not the right choice - ARG is the right choice | 22:07 |
| clarkb | dmsimard: the first time yes, beacuse there is no delta yet. But then the second time we see there are 6 more dicts for example | 22:08 |
| clarkb | and one more stringio | 22:08 |
| dmsimard | oh | 22:09 |
| corvus | they were not far apart; if you do it 2 more times you'll get better deltas | 22:09 |
| corvus | (always run it twice to turn off yappi) | 22:09 |
| clarkb | we probably want one after running executor for like half an hour | 22:10 |
| clarkb | that gives us a baseline on function counts | 22:10 |
| clarkb | then one a day later or so? | 22:10 |
| clarkb | I'm reruning sigusr2 on ze01 now | 22:16 |
| mordred | corvus, clarkb: I build image on the held node using the above dockerfile approach and the build was successful | 22:17 |
| mordred | obviously the proof will be in the job going green | 22:17 |
| *** yamamoto has quit IRC | 22:18 | |
| mordred | corvus: I think we should make build_arg a list - --build-arg can be passed to docker build more than one time | 22:19 |
| clarkb | corvus: dmsimard http://paste.openstack.org/show/742357/ is the delta between corvus' second usr2 and my first (I did run it twice but the delta between my first an second is less interesting) | 22:19 |
| clarkb | is it possible we are leaking functions somehow? | 22:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Delete change tags from docker image repos https://review.openstack.org/630782 | 22:20 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid master: Added configuration for MYSQL SSL connection https://review.openstack.org/595839 | 22:21 |
| corvus | clarkb: hrm. now i sort of want to add the repl to the executor | 22:22 |
| corvus | see what those function objects are | 22:22 |
| dmsimard | so it's just "kill -SIGUSR2 $pid" once to start yappi then another time to stop it ? | 22:22 |
| corvus | clarkb: i think i like your 30 minutes / 24 hour idea | 22:22 |
| corvus | maybe we should do a full zuul restart (i need a scheduler change anyway) and do that | 22:23 |
| corvus | i'm worried that if we wait 24 hours now, the leak (if any) will be too slow to jump out at us | 22:23 |
| clarkb | dmsimard: yes, though I waited for the finish message to show up in the log message before running the second time (I don't think this is necessary though) | 22:24 |
| clarkb | corvus: ++ | 22:24 |
| mordred | corvus: if we do a full restart - it would be nice to get the git driver sources for gerrit things in | 22:24 |
| mordred | corvus: but also, if you don't want to wait on that, totally fine | 22:24 |
| corvus | i need to run an errand; maybe clarkb could review https://review.openstack.org/630740 and i can restart when i get back? | 22:25 |
| *** smarcet has quit IRC | 22:26 | |
| clarkb | reviewing now | 22:26 |
| mordred | clarkb: https://review.openstack.org/#/c/630744/ when you're done reviewing that | 22:26 |
| *** yamamoto has joined #openstack-infra | 22:26 | |
| clarkb | I've approved the first and +2'd the second (I blieve the second needs gerrit to be restarted before it can't be applied) | 22:27 |
| dmsimard | clarkb: ah, I wanted to compare numbers with the software factory deployment but we're not carrying yappi right now :( | 22:29 |
| corvus | dmsimard: these numbers are from objgraph, not yappi | 22:29 |
| dmsimard | yappi is required though, right ? | 22:30 |
| corvus | i don't think so? | 22:30 |
| clarkb | no yappi does a different set of data | 22:30 |
| corvus | verified: they are both optional | 22:30 |
| clarkb | reading the code we check for each independently | 22:30 |
| dmsimard | ah but we don't have objgraph either | 22:30 |
| * dmsimard adds to todo list | 22:31 | |
| corvus | (it's just that we *do* have yappi installed, so we need to run it twice since the same event starts/stops yappi) | 22:31 |
| *** yamamoto has quit IRC | 22:31 | |
| *** rcernin has joined #openstack-infra | 22:31 | |
| corvus | all of these things could probably be migrated to command socket commands now | 22:31 |
| *** yamamoto has joined #openstack-infra | 22:32 | |
| *** yamamoto has quit IRC | 22:36 | |
| *** yamamoto has joined #openstack-infra | 22:37 | |
| *** ekultails has quit IRC | 22:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 22:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 22:44 |
| openstackgerrit | Trevor McCasland proposed openstack-infra/subunit2sql master: Fix 'Content' object is not iterable https://review.openstack.org/627409 | 22:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 22:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 22:48 |
| mordred | corvus: ok. there's a stab at it being a list. | 22:48 |
| *** yamamoto has quit IRC | 22:49 | |
| *** sreejithp has quit IRC | 22:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add gerrit.googlesource.com to zuul using git driver https://review.openstack.org/630740 | 22:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Add CentOS NetworkManager testing node" https://review.openstack.org/629681 | 22:58 |
| *** smarcet has joined #openstack-infra | 23:00 | |
| *** yamamoto has joined #openstack-infra | 23:01 | |
| *** boden has quit IRC | 23:03 | |
| *** smarcet has quit IRC | 23:04 | |
| *** mriedem is now known as mriedem_away | 23:07 | |
| *** efried has quit IRC | 23:08 | |
| *** rascasoft has quit IRC | 23:10 | |
| clarkb | I just finished doing it myself so may as well remind others to do so too, openstack board elections happening now, if you are an individual member you should go and vote :) | 23:10 |
| *** yamamoto has quit IRC | 23:11 | |
| mordred | you also get the opportunity to vote on bylaws changes! | 23:11 |
| *** yamamoto has joined #openstack-infra | 23:11 | |
| clarkb | mordred: can I get you to review https://review.openstack.org/#/c/630739/ and https://review.openstack.org/#/c/630742/ ? | 23:14 |
| mordred | clarkb: looking | 23:15 |
| *** yamamoto has quit IRC | 23:15 | |
| *** efried has joined #openstack-infra | 23:15 | |
| mordred | clarkb: +2 on both | 23:16 |
| clarkb | thank you | 23:16 |
| clarkb | any other infra roots want to take a look? I'm running out of time today where I can focus and pay attention to those changes going in but would be good to get them in soon if we can | 23:16 |
| clarkb | fungi: ^ you probably most up to date on that stuff if you are still around | 23:17 |
| mordred | clarkb: there's a few more plugins in the plugins dir than I have in the plugin build instructions in the dockerfile | 23:17 |
| mordred | clarkb: for gerrit | 23:17 |
| *** yamamoto has joined #openstack-infra | 23:17 | |
| clarkb | mordred: some of them are bundled like replication iirc | 23:18 |
| clarkb | the code is organized as a plugin but they get automagically built for you I think, we should be able to double check that with your builds? | 23:18 |
| mordred | clarkb: singleusergroup, reviewnotes, commit-message-length-validator, download-commands | 23:18 |
| corvus | i'm back; looks like the googlesource source change hasn't landed on zuul01 yet | 23:19 |
| mordred | ah - good point | 23:19 |
| mordred | corvus: it just merged at 22:58:05 | 23:19 |
| mordred | so it should be there soon | 23:19 |
| fungi | clarkb: yeah, still writing that zuul spotlight for the osf newsletter, but will take a look | 23:19 |
| corvus | yep; i'll check back in a bit | 23:19 |
| corvus | ah drat | 23:20 |
| corvus | i forgot to change a dockerfile to trigger a build | 23:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Delete change tags from docker image repos https://review.openstack.org/630782 | 23:21 |
| *** rascasoft has joined #openstack-infra | 23:21 | |
| clarkb | mordred: ya check gerrit/plugins | 23:21 |
| clarkb | mordred: contains BUILD commit-message-length-validator download-commands external_plugin_deps.bzl hooks replication reviewnotes singleusergroup | 23:21 |
| *** yamamoto has quit IRC | 23:21 | |
| mordred | ah - cool | 23:21 |
| fungi | clarkb: the change being reverted, was that one zaro added because we needed an offline reindex upgrading from 2.13? | 23:21 |
| clarkb | fungi: ianw added it. I'm not sure why. It shouldn't have been necessary for any of our servers as they were all initially installed long ago | 23:22 |
| clarkb | fungi: that code was specifically there to handle the first ever install you make | 23:22 |
| fungi | add, it's been there since august 2016 | 23:22 |
| clarkb | (per the comment) | 23:22 |
| fungi | https://review.openstack.org/359683 | 23:23 |
| fungi | cool, thanks! | 23:23 |
| clarkb | my first ps failed because of the thing that fixes | 23:23 |
| clarkb | I then had to update the rspec tests to set offline_reindex to true to get the initial install to reindex | 23:23 |
| ianw | in my notes around that day i have " migrate ~gerrit2 on review-dev to new cinder volume" | 23:24 |
| *** rlandy has quit IRC | 23:25 | |
| clarkb | fungi: thanks. Those should both be noops for running gerrit (which I'll watch to confirm) then tomorrow pick up the test of upgrading on review-dev again | 23:25 |
| ianw | clarkb: ahh, here's what i was doing -> http://lists.openstack.org/pipermail/openstack-infra/2016-August/004684.html | 23:25 |
| clarkb | ianw: I'm guessing that puppet ran first and gerrit failed to start before you copied things over | 23:26 |
| clarkb | since launch node ran puppet at the time | 23:26 |
| *** rascasoft has quit IRC | 23:26 | |
| clarkb | setting offline_reindex to true should work for that case if we run into it today | 23:26 |
| mordred | 2016 was so long ago | 23:27 |
| *** yamamoto has joined #openstack-infra | 23:30 | |
| corvus | clarkb, dmsimard: have we just been assuming that the executor queue is related to swap? | 23:31 |
| corvus | clarkb, dmsimard: i just looked at the logs for ze01, and the proximate cause for unregistering has consistently been 'too many starting builds'. | 23:32 |
| corvus | that could be indirectly related to swap | 23:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 23:33 |
| *** yamamoto has quit IRC | 23:33 | |
| *** dkehn has left #openstack-infra | 23:33 | |
| clarkb | corvus: I thought you or pabelanger? had tied it to swap back in december | 23:34 |
| clarkb | I haven't confirmed that recently | 23:34 |
| ianw | ^ while dogpile.cache maybe isn't the most interesting given it's general stability, it's the first example of pulling in github repos in devstack testing | 23:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 23:35 |
| *** yamamoto has joined #openstack-infra | 23:35 | |
| mordred | ianw: also, dogpile.cache 0.7.0 broke us via openstacksdk :) | 23:35 |
| ianw | so it would probably be cool for dogpile.cache to integrate that ... although i imagine anyone upstream actually debugging it would be quite an effort | 23:35 |
| ianw | mordred: yep, that's what prompted me to see if we could pull it from master | 23:36 |
| mordred | ++ | 23:36 |
| mordred | MAYBE I'll get a green image build today | 23:36 |
| *** rascasoft has joined #openstack-infra | 23:36 | |
| *** yamamoto has quit IRC | 23:37 | |
| *** eernst has quit IRC | 23:39 | |
| *** dkehn has joined #openstack-infra | 23:41 | |
| *** rascasoft has quit IRC | 23:41 | |
| *** signed8bit has quit IRC | 23:41 | |
| *** yamamoto has joined #openstack-infra | 23:43 | |
| *** rfolco has quit IRC | 23:45 | |
| *** rfolco has joined #openstack-infra | 23:45 | |
| *** yamamoto has quit IRC | 23:46 | |
| *** rascasoft has joined #openstack-infra | 23:47 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Set java lib user ownership to gerrit2 https://review.openstack.org/630739 | 23:48 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Revert "Create index on fresh install" https://review.openstack.org/630742 | 23:48 |
| corvus | googlesource change is on disk | 23:49 |
| mordred | corvus: https://review.openstack.org/#/c/630468/16/playbooks/zuul/build-image/run.yaml <-- I tried moving this to a wrapped block - the previous patch didn't get indentation and blank lines right | 23:49 |
| mordred | corvus: reasoning about the intersection of jinja and yaml sometimes hurts my head | 23:50 |
| mordred | oh - I guess I shoud have updated the build_args patch, not that one | 23:50 |
| corvus | heh, mildly confusing :) | 23:50 |
| corvus | mordred: still failing: http://logs.openstack.org/68/630468/16/check/system-config-build-image-gerrit/b29aa22/ara-report/result/cfdf744c-abc2-42dd-b31b-a9006dd15c31/ | 23:51 |
| *** rascasoft has quit IRC | 23:52 | |
| ianw | fungi: i'd be interested if you have any thoughts on mixing pypi install jobs and git install jobs and bindep.txt. in short, when installing from git we can use the packages bindep, but when installing releases from pypi we have no insight into that. i'm trying to think how we can cover both without too much duplication | 23:52 |
| ianw | fungi: to be exact see https://review.openstack.org/#/c/626068/ | 23:53 |
| corvus | mordred: you turned '--tag' into '-tag' | 23:53 |
| fungi | ianw: is this for devstack, i guess? | 23:53 |
| fungi | good guess! | 23:53 |
| corvus | mordred: not sure if that's _the_ issue, but maybe _an_ issue | 23:53 |
| mordred | corvus: thats likely because I can't operate editors | 23:53 |
| clarkb | its a bsd tool now | 23:53 |
| ianw | fungi: yep; i'm thinking that we just have to let plugins (like nodepool) have a fallback bindep, that duplicates what's in the project's it may install's bindep | 23:54 |
| corvus | clarkb: docker was probably written on a mac, so... surprised it wasn't like that all along | 23:54 |
| fungi | ianw: it's a good question. bindep's original design as a helper for setting up development environments sort of assumed you had the source code handy for the software on which you were going to do development work | 23:54 |
| *** yamamoto has joined #openstack-infra | 23:55 | |
| corvus | i'm going to restart all of zuul now | 23:55 |
| fungi | ianw: for devstack, i envisioned just converting its package list into a bindep.txt (so presumably plugins could also provide their own additions too?) | 23:55 |
| fungi | ianw: i didn't consider trying to reuse each project's bindep list to build one for devstack, if that helps | 23:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add support for per-image build-arg settings https://review.openstack.org/630777 | 23:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Build a gerrit image https://review.openstack.org/630468 | 23:56 |
| mordred | corvus: maybe this time is the charm | 23:56 |
| fungi | ianw: which packages you need to install in a nova development environment won't necessarily be a 1:1 match with what packages devstack needs to be able to start the nova service | 23:56 |
| ianw | fungi: yeah, that's were i'm at. so for example, with nodepool's plugin, it's bindep.txt will need to duplicate dib's bindep.txt to get debootstrap etc etc | 23:56 |
| ianw | but in the -src jobs, we can reference dib's bindep directly | 23:57 |
| *** yamamoto has quit IRC | 23:57 | |
| fungi | it's an intruiging optimization i hadn't considered | 23:57 |
| *** yamamoto has joined #openstack-infra | 23:57 | |
| fungi | though other than avoiding some duplication, i'm not convinced the added complexity is warranted | 23:58 |
| ianw | so now it becomes a mess ... how does devstack know what not to use when it's a -src job v a released job | 23:58 |
| *** markvoelker has joined #openstack-infra | 23:58 | |
| fungi | i mean, it's possible you might want different/additional packages installed in a dib development environment than in a devstack deployment using the nodepool devstack plugin | 23:58 |
| fungi | so just always keeping those lists separate has its advantages in simplicity | 23:59 |
| fungi | even if it means needing to remember to add some of the dependencies in both places | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!