| *** bobh has joined #openstack-infra | 00:06 | |
| *** gyee has joined #openstack-infra | 00:08 | |
| *** lbragstad has quit IRC | 00:10 | |
| *** gyee has quit IRC | 00:12 | |
| ianw | clarkb: thanks, seems there was desire to do that. think it will require manual cleanup but that's ok | 00:30 |
|---|---|---|
| clarkb | ya likely need to clean out the old images once we stop using the node type | 00:30 |
| *** openstackgerrit has quit IRC | 00:34 | |
| *** bobh has quit IRC | 00:36 | |
| clarkb | mrhillsman: got around to reviewing https://review.openstack.org/#/c/626084/1 there is a functionality issue I've called out inline with some higher level things we should double check on the top level comment | 00:38 |
| *** jamesmcarthur has joined #openstack-infra | 00:38 | |
| clarkb | I don't know why but the spam in -meeting made me lol | 00:39 |
| *** eernst has joined #openstack-infra | 00:41 | |
| clarkb | any othe rinfra-root want to review https://review.openstack.org/#/c/629439/1 to remove unnused meeting channels from meetbot? | 00:42 |
| *** jamesmcarthur has quit IRC | 00:43 | |
| *** eernst has quit IRC | 00:44 | |
| *** eernst_ has joined #openstack-infra | 00:44 | |
| *** eernst_ has quit IRC | 00:45 | |
| *** eernst has joined #openstack-infra | 00:45 | |
| *** eernst has quit IRC | 00:49 | |
| *** openstackgerrit has joined #openstack-infra | 00:54 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 00:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Add CentOS NetworkManager testing node https://review.openstack.org/629680 | 00:54 |
| *** hwoarang_ has joined #openstack-infra | 00:57 | |
| *** hwoarang has quit IRC | 00:59 | |
| *** gyee has joined #openstack-infra | 01:01 | |
| *** tosky has quit IRC | 01:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use promote pipeline for system-config images https://review.openstack.org/629957 | 01:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use promote pipeline for system-config images https://review.openstack.org/629957 | 01:07 |
| *** rlandy has quit IRC | 01:07 | |
| *** bobh has joined #openstack-infra | 01:09 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use promote pipeline for system-config images https://review.openstack.org/629957 | 01:09 |
| *** yamamoto has quit IRC | 01:10 | |
| *** armax has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has joined #openstack-infra | 01:12 | |
| *** sthussey has quit IRC | 01:12 | |
| *** yamamoto has joined #openstack-infra | 01:14 | |
| *** markvoelker has joined #openstack-infra | 01:15 | |
| *** armax has quit IRC | 01:15 | |
| *** wolverineav has quit IRC | 01:20 | |
| *** wolverineav has joined #openstack-infra | 01:21 | |
| *** ruffian_sheep has joined #openstack-infra | 01:26 | |
| *** wolverineav has quit IRC | 01:28 | |
| *** armax has joined #openstack-infra | 01:29 | |
| *** bobh has quit IRC | 01:31 | |
| *** bobh has joined #openstack-infra | 01:32 | |
| *** yamamoto has quit IRC | 01:35 | |
| *** gyee has quit IRC | 01:37 | |
| *** bobh has quit IRC | 01:37 | |
| *** gyee has joined #openstack-infra | 01:38 | |
| ruffian_sheep | Does anyone know how to set the piplines of jenkins jobs build uesd in zuul | 01:40 |
| fungi | ruffian_sheep: i'm not sure what you mean. the current major version of zuul doesn't come with any jenkins integration | 01:44 |
| fungi | ruffian_sheep: also, if you're trying to run zuul, the #zuul channel is likely a more appropriate place to ask questions about it | 01:45 |
| *** gyee has quit IRC | 01:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:49 | |
| *** armax has quit IRC | 01:50 | |
| *** jamesmcarthur has quit IRC | 01:56 | |
| *** bhavikdbavishi has quit IRC | 01:59 | |
| *** jamesmcarthur has joined #openstack-infra | 01:59 | |
| *** jamesmcarthur has quit IRC | 02:04 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| ruffian_sheep | Thanks for your reply!I don't know how to send the message privately ;p | 02:07 |
| ruffian_sheep | fungi | 02:07 |
| fungi | there's no need to send me a message privately, we appreciate open public discussion here | 02:12 |
| *** sshnaidm is now known as sshnaidm|off | 02:14 | |
| *** yamamoto has joined #openstack-infra | 02:16 | |
| ruffian_sheep | No,maybe something I said isn't clearly.I can see my nickname after your.Like that <fungi> ruffian_sheep: i'm not sure what you mean. the current major version of zuul doesn't come with any jenkins integration | 02:16 |
| *** jamesmcarthur has quit IRC | 02:17 | |
| ruffian_sheep | And your nickname is red,so I can see your reply clearly.Is my first time to use irc.I don't know how to do it so that you can notice my message;) | 02:18 |
| fungi | ahh, just mentioning my nick anywhere in your message causes my client to highlight it | 02:18 |
| *** jamesmcarthur has joined #openstack-infra | 02:19 | |
| ruffian_sheep | fungi fungi fungi,is my nickname is red now? | 02:21 |
| fungi | yep | 02:22 |
| fungi | well, in my client you're yellow with a magenta background, but highlighted yes | 02:22 |
| ruffian_sheep | Waoooooo,get it ;P | 02:22 |
| *** jamesmcarthur has quit IRC | 02:23 | |
| *** yamamoto has quit IRC | 02:24 | |
| *** smarcet has joined #openstack-infra | 02:25 | |
| ruffian_sheep | OK,get a new skill lolo | 02:26 |
| mrhillsman | clarkb: ack | 02:30 |
| *** jamesmcarthur has joined #openstack-infra | 02:31 | |
| *** bobh has joined #openstack-infra | 02:33 | |
| *** bobh has quit IRC | 02:38 | |
| *** yamamoto has joined #openstack-infra | 02:39 | |
| *** markvoelker has quit IRC | 02:42 | |
| *** jamesmcarthur has quit IRC | 02:48 | |
| *** psachin has joined #openstack-infra | 02:50 | |
| *** lbragstad has joined #openstack-infra | 02:52 | |
| *** markvoelker has joined #openstack-infra | 02:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use promote pipeline for system-config images https://review.openstack.org/629957 | 02:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use a mix-in job to reduce duplication https://review.openstack.org/629983 | 02:58 |
| corvus | clarkb, ttx: +2 629439 but didn't approve in case we want to tie that to setting the topic | 03:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use a mix-in job to reduce duplication https://review.openstack.org/629983 | 03:02 |
| *** whoami-rajat has joined #openstack-infra | 03:03 | |
| *** hwoarang has joined #openstack-infra | 03:09 | |
| *** hwoarang_ has quit IRC | 03:09 | |
| *** wolverineav has joined #openstack-infra | 03:11 | |
| ruffian_sheep | <openstackgerrit>:Is this the answer to set the piplines used in zuul? | 03:17 |
| *** _ari_ has quit IRC | 03:21 | |
| ruffian_sheep | ...Is this a robot..? | 03:25 |
| *** markvoelker has quit IRC | 03:28 | |
| *** bobh has joined #openstack-infra | 03:38 | |
| *** bobh has quit IRC | 03:43 | |
| *** Haunted330 has joined #openstack-infra | 03:44 | |
| *** ykarel|away has joined #openstack-infra | 03:46 | |
| *** markvoelker has joined #openstack-infra | 03:48 | |
| Haunted330 | This channel closes in 6 minutes. | 03:54 |
| *** yamamoto has quit IRC | 03:55 | |
| *** ykarel|away has quit IRC | 04:07 | |
| *** yamamoto has joined #openstack-infra | 04:09 | |
| Haunted330 | This channel is closed. You will have to leave. | 04:11 |
| *** Haunted330 has quit IRC | 04:12 | |
| *** yamamoto has quit IRC | 04:13 | |
| *** udesale has joined #openstack-infra | 04:14 | |
| *** ykarel has joined #openstack-infra | 04:15 | |
| *** ykarel_ has joined #openstack-infra | 04:18 | |
| *** ykarel has quit IRC | 04:21 | |
| *** markvoelker has quit IRC | 04:26 | |
| *** ykarel_ has quit IRC | 04:28 | |
| *** ykarel has joined #openstack-infra | 04:32 | |
| *** lbragstad has quit IRC | 04:35 | |
| *** ykarel has quit IRC | 04:39 | |
| *** efried1 has joined #openstack-infra | 04:39 | |
| *** lpetrut has joined #openstack-infra | 04:39 | |
| *** efried has quit IRC | 04:40 | |
| *** efried1 is now known as efried | 04:40 | |
| *** ykarel has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has joined #openstack-infra | 04:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:51 | |
| *** jamesmcarthur has quit IRC | 04:53 | |
| *** wolverineav has quit IRC | 04:58 | |
| *** markvoelker has joined #openstack-infra | 05:05 | |
| *** ykarel has quit IRC | 05:05 | |
| *** lpetrut has quit IRC | 05:08 | |
| anteaya | ruffian_sheep: yes openstackgerrit is a robot | 05:08 |
| anteaya | ruffian_sheep: you were speaking with fungi earlier, that person is human | 05:09 |
| *** markvoelker has quit IRC | 05:11 | |
| *** markvoelker has joined #openstack-infra | 05:11 | |
| *** markvoelker has quit IRC | 05:15 | |
| *** markvoelker has joined #openstack-infra | 05:16 | |
| *** yboaron_ has joined #openstack-infra | 05:25 | |
| *** ykarel has joined #openstack-infra | 05:26 | |
| *** dave-mccowan has quit IRC | 05:28 | |
| *** yboaron_ has quit IRC | 05:29 | |
| *** yboaron_ has joined #openstack-infra | 05:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add version to info endpoint https://review.openstack.org/609571 | 05:35 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add about dropdown to display zuul version https://review.openstack.org/630027 | 05:35 |
| *** yboaron_ has quit IRC | 05:36 | |
| *** chkumar|out is now known as chandankumar | 05:40 | |
| ruffian_sheep | <anteaya>:Interesting misunderstanding | 05:49 |
| *** rcernin has quit IRC | 05:58 | |
| *** kjackal has joined #openstack-infra | 06:00 | |
| ruffian_sheep | <jiapei>:I have done it before ,just don't know how to use the layout.yaml of zuul and the project.yaml of jenkins_jobs. | 06:04 |
| *** wolverineav has joined #openstack-infra | 06:04 | |
| *** rcernin has joined #openstack-infra | 06:04 | |
| *** smarcet has quit IRC | 06:06 | |
| *** wolverineav has quit IRC | 06:09 | |
| chandankumar | AJaeger: Hello | 06:14 |
| chandankumar | AJaeger: in this review https://review.openstack.org/#/c/627482/ I am trying to run os_tempest with devstack | 06:15 |
| chandankumar | AJaeger: but on setting tempest_service_setup_host: 'tempest' the zuul is giving retry_error on this job | 06:15 |
| chandankumar | AJaeger: please have a look, thanks :-) | 06:16 |
| jiapei | ruffian_sheep: Here is the link to set project.yaml https://zuul-ci.org/docs/zuul/user/config.html#project, but I didn't find a related link on the layout.yaml you said... | 06:18 |
| *** markvoelker has quit IRC | 06:20 | |
| *** e0ne has joined #openstack-infra | 06:24 | |
| ruffian_sheep | <jiapei>:I learn how to set the CI by this.https://docs.openstack.org/infra/openstackci/third_party_ci.html The layout.yaml path is /etc/zuul/layout.yaml | 06:26 |
| jiapei | ruffian_sheep> That link is too old, it uses Zuul v2 and Jenkins, but the community is using Zuul V3 now to set up 3rd CI. | 06:29 |
| *** markvoelker has joined #openstack-infra | 06:31 | |
| *** lifeless_ is now known as lifeless | 06:36 | |
| *** markvoelker has quit IRC | 06:36 | |
| *** e0ne has quit IRC | 06:36 | |
| jiapei | fungi: I can write a document on how to setup a 3rd party CI with Zuul v3 based on the experience I had during setting up Ironic 3rd party CI, but I don't know where I can put the document to. Most 3rd party builder will follow this link https://docs.openstack.org/infra/openstackci/third_party_ci.html, but obviously it is too old. Could you help point to a link where I can write the document? | 06:37 |
| ruffian_sheep | <jiapei>:I have asked people in the contact information,but no one reply:( | 06:39 |
| *** kjackal has quit IRC | 06:39 | |
| jiapei | ruffian_sheep: They are asleep this time. But why do you need to setup a Zuul v2 based CI? The Zuul v3 will be easier... | 06:42 |
| *** e0ne has joined #openstack-infra | 06:45 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add buildset uuid https://review.openstack.org/630034 | 06:45 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route https://review.openstack.org/630035 | 06:45 |
| ruffian_sheep | I don't know...Just following the guidance document | 06:45 |
| jiapei | ruffian_sheep: you will only to setup a node (VM or baremetal) with Zuul and Nodepool for the CI infrastructure and another node to act as the test machine | 06:45 |
| ruffian_sheep | jiapei | 06:45 |
| jiapei | The guidance document needs updating | 06:46 |
| jiapei | Which driver do you need to test with CI? ruffian_sheep | 06:47 |
| ruffian_sheep | <jiapei>:Actually,it told me to use jenkins,zuul,Nodepool and so on to do it | 06:47 |
| ruffian_sheep | <jiapei>:Cinder | 06:47 |
| ruffian_sheep | <jiapei>:https://review.openstack.org/#/c/612311/ Jay Bryant asked me to build a 3rd Party CI for this. | 06:49 |
| ruffian_sheep | <jiapei>:I have not any knowledge about that before | 06:50 |
| jiapei | ruffian_sheep: I understand, I also tried to setup a CI before | 06:50 |
| *** roman_g has quit IRC | 06:51 | |
| jiapei | My advice is that you can try this link to get a general understanding of the CI infrastructure https://zuul-ci.org/docs/zuul/ | 06:52 |
| jiapei | You probably only need Zuul and Nodepool, and maybe some plugin for Cinder (like molteniron for Ironic) | 06:54 |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 06:55 |
| *** Tengu has quit IRC | 06:57 | |
| *** markvoelker has joined #openstack-infra | 07:02 | |
| *** pcaruana has joined #openstack-infra | 07:05 | |
| *** rcernin has quit IRC | 07:06 | |
| *** kopecmartin|off is now known as kopecmartin | 07:07 | |
| *** markvoelker has quit IRC | 07:07 | |
| *** markvoelker has joined #openstack-infra | 07:08 | |
| *** Tengu has joined #openstack-infra | 07:08 | |
| *** bhavikdbavishi has quit IRC | 07:10 | |
| *** markvoelker has quit IRC | 07:13 | |
| *** gfidente has joined #openstack-infra | 07:15 | |
| *** quiquell|off is now known as quiquell | 07:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add buildset uuid column https://review.openstack.org/630034 | 07:17 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route https://review.openstack.org/630035 | 07:17 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 07:17 |
| *** e0ne has quit IRC | 07:27 | |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| *** e0ne has joined #openstack-infra | 07:36 | |
| *** e0ne has quit IRC | 07:36 | |
| *** aojea has joined #openstack-infra | 07:41 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add buildset uuid column https://review.openstack.org/630034 | 07:46 |
| *** dpawlik has joined #openstack-infra | 07:47 | |
| *** psachin has quit IRC | 07:47 | |
| *** rpittau has joined #openstack-infra | 07:48 | |
| *** slaweq has joined #openstack-infra | 07:49 | |
| *** slaweq has quit IRC | 07:57 | |
| *** wolverineav has joined #openstack-infra | 07:58 | |
| *** ccamacho has joined #openstack-infra | 07:59 | |
| *** e0ne has joined #openstack-infra | 08:01 | |
| *** wolverineav has quit IRC | 08:02 | |
| AJaeger | chandankumar: follow the logs in zuul.openstack.org for your change and see where it fails... I can't help further | 08:04 |
| *** roman_g has joined #openstack-infra | 08:07 | |
| *** slaweq has joined #openstack-infra | 08:08 | |
| *** ginopc has joined #openstack-infra | 08:08 | |
| *** diablo_rojo has quit IRC | 08:08 | |
| *** e0ne has quit IRC | 08:10 | |
| *** markvoelker has joined #openstack-infra | 08:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route https://review.openstack.org/630035 | 08:17 |
| *** e0ne has joined #openstack-infra | 08:20 | |
| *** e0ne has quit IRC | 08:21 | |
| *** jbadiapa has joined #openstack-infra | 08:22 | |
| *** ykarel|lunch is now known as ykarel | 08:23 | |
| *** markvoelker has quit IRC | 08:23 | |
| *** psachin has joined #openstack-infra | 08:32 | |
| *** ginopc has quit IRC | 08:35 | |
| *** ginopc has joined #openstack-infra | 08:35 | |
| *** tosky has joined #openstack-infra | 08:40 | |
| *** kjackal has joined #openstack-infra | 08:43 | |
| *** stakeda has quit IRC | 08:43 | |
| *** jamesmcarthur has joined #openstack-infra | 08:50 | |
| *** jamesmcarthur has quit IRC | 08:54 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** panda|off is now known as panda | 08:55 | |
| *** jpich has joined #openstack-infra | 08:57 | |
| *** jrist has quit IRC | 09:05 | |
| *** jrist has joined #openstack-infra | 09:06 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route https://review.openstack.org/630035 | 09:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 09:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 09:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page https://review.openstack.org/630079 | 09:07 |
| *** shardy has joined #openstack-infra | 09:20 | |
| *** gfidente has quit IRC | 09:22 | |
| *** rascasoft has joined #openstack-infra | 09:23 | |
| *** gfidente has joined #openstack-infra | 09:27 | |
| ruffian_sheep | <jiapei>:If this is not a company secret, can you give me the configuration file that you used to configure the CI environment before? | 09:34 |
| ssbarnea|rover | infra-root: i seen an outofspace error last night with finger://ze08.openstack.org/3f8d1e1f50c446d081b9838b4faeccd3 -- who can take care of this? or is automatic. | 09:43 |
| ssbarnea|rover | what makes it even worse is that the zuul comment on this translates in a message full of green sucess and most people do not realise that something (or what) was wrong. | 09:46 |
| ssbarnea|rover | arxcruz|ruck: see https://s3.sbarnea.com/ss/190111-Change_Ia5bcd556_GATE_CHECK_for_TripleO__review.openstack_Code_Review_.png | 09:47 |
| ssbarnea|rover | the table view in zuul shows only succeses, but the vote was -1 | 09:47 |
| *** jangutter has quit IRC | 09:48 | |
| *** derekh has joined #openstack-infra | 09:48 | |
| panda | AJaeger: a files: configuration in zuul will override the parent or merge the lists ? | 09:54 |
| arxcruz|ruck | fungi: around? we get a tripleo-ci-centos-7-scenario001-multinode-oooq-container finger://ze08.openstack.org/3f8d1e1f50c446d081b9838b4faeccd3 : DISK_FULL in 2h 43m 39s error | 09:55 |
| ssbarnea|rover | is there a a finger handler extension for firefox? (google didn't help much with finger...ing) | 10:00 |
| *** e0ne has joined #openstack-infra | 10:01 | |
| *** smarcet has joined #openstack-infra | 10:01 | |
| *** jtomasek has quit IRC | 10:03 | |
| AJaeger | panda: Please read https://zuul-ci.org/docs/zuul/user/config.html#attr-job.files - it's overwritten | 10:03 |
| *** smarcet has quit IRC | 10:04 | |
| panda | AJaeger: I read it three times, wasn't still clear to me, sorry. | 10:07 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix test_load_governor on large machines https://review.openstack.org/630118 | 10:08 |
| *** kjackal has quit IRC | 10:10 | |
| openstackgerrit | Andriy Shevchenko proposed openstack/gertty master: Updatae home-page https://review.openstack.org/630120 | 10:11 |
| *** shardy has quit IRC | 10:12 | |
| AJaeger | panda: if it wasn't clear - do you have suggestions on how to improve the wording for the next person? The document lives in the zuul repo, patches are always welcome to improve docs! | 10:14 |
| *** shardy has joined #openstack-infra | 10:15 | |
| panda | AJaeger: in these cases I usually consider I'm just dumb. | 10:16 |
| openstackgerrit | Arnaud Morin proposed openstack-infra/project-config master: Enable back GRA1 on OVH cloud https://review.openstack.org/630130 | 10:19 |
| stephenfin | AJaeger: Is this something that could be shuttered? https://github.com/openstack/gantt | 10:21 |
| stephenfin | Just saw a review coming up for it on #openstack-nova but the README says it's dead for quite some time | 10:21 |
| AJaeger | stephenfin: sure - see https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project | 10:23 |
| * stephenfin clicks | 10:23 | |
| *** ramishra has joined #openstack-infra | 10:23 | |
| AJaeger | stephenfin: note you can always revive a repo... | 10:24 |
| AJaeger | stephenfin: I suggest to retire python-ganttclient together with gantt | 10:24 |
| stephenfin | AJaeger: agreed | 10:24 |
| stephenfin | Looks like it never made it into requirements either. Good | 10:24 |
| * AJaeger is happy to guide you through the process... | 10:24 | |
| stephenfin | AJaeger: Let me follow that doc. Assuming there's nothing else needed, the only other thing I'll need is for someone else to do the retirement announcement. Probably melwitt, I'd say | 10:25 |
| stephenfin | (as nova PTL) | 10:25 |
| *** shardy_ has joined #openstack-infra | 10:26 | |
| AJaeger | stephenfin: and it has noop-jobs, so you only need one change for project-config like in https://review.openstack.org/#/c/628850/ | 10:26 |
| AJaeger | stephenfin: best to discuss with PTL first | 10:27 |
| stephenfin | Agreed. I won't post anything until they agree | 10:27 |
| stephenfin | But now I know the process. Should be a no-brainer | 10:27 |
| AJaeger | yeah | 10:27 |
| *** shardy has quit IRC | 10:29 | |
| stephenfin | AJaeger: What about channels? Looks like there was an openstack-gantt channel at one time | 10:30 |
| stephenfin | Ditto for murano | 10:30 |
| stephenfin | ...which is currently empty | 10:30 |
| AJaeger | remove those channels | 10:31 |
| stephenfin | And last one: the old gerrit acl config files in 'gerrit/acls/openstack' (project-config) | 10:31 |
| stephenfin | Any reason to keep those? | 10:32 |
| *** yboaron_ has joined #openstack-infra | 10:32 | |
| AJaeger | no, we delete them -and use acl-config: retired.config basically | 10:32 |
| stephenfin | ack | 10:32 |
| *** yboaron_ has quit IRC | 10:36 | |
| *** shardy_ has quit IRC | 10:36 | |
| *** kjackal has joined #openstack-infra | 10:36 | |
| *** ruffian_sheep has quit IRC | 10:37 | |
| *** electrofelix has joined #openstack-infra | 10:48 | |
| *** shardy has joined #openstack-infra | 10:49 | |
| *** ykarel_ has joined #openstack-infra | 10:51 | |
| *** ramishra has quit IRC | 10:52 | |
| *** ykarel has quit IRC | 10:54 | |
| *** udesale has quit IRC | 10:54 | |
| *** psachin has quit IRC | 10:55 | |
| *** psachin has joined #openstack-infra | 10:57 | |
| *** gfidente has quit IRC | 10:57 | |
| *** shardy has quit IRC | 10:58 | |
| *** ykarel has joined #openstack-infra | 11:02 | |
| *** ykarel_ has quit IRC | 11:03 | |
| *** gfidente has joined #openstack-infra | 11:03 | |
| *** kjackal has quit IRC | 11:10 | |
| *** kjackal_v2 has joined #openstack-infra | 11:10 | |
| *** ramishra has joined #openstack-infra | 11:14 | |
| *** dims has quit IRC | 11:21 | |
| *** dtantsur|afk is now known as dtantsur | 11:26 | |
| *** wolverineav has joined #openstack-infra | 11:34 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Retire gantt, python-ganttclient https://review.openstack.org/630162 | 11:34 |
| *** wolverineav has quit IRC | 11:38 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix test_load_governor on large machines https://review.openstack.org/630118 | 11:39 |
| *** lpetrut has joined #openstack-infra | 11:43 | |
| frickler | amorin: did you also do a quota resync for gra1? we still have some zombie servers listed there http://paste.openstack.org/show/741771/ | 11:47 |
| frickler | not sure whether nodepool will see these and reduce the quota it tries to use | 11:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable back GRA1 on OVH cloud https://review.openstack.org/630130 | 11:50 |
| frickler | ah, they show up in state deleting here, so probably should be fine grafana.openstack.org/d/rZtIH5Imz/nodepool | 11:50 |
| frickler | infra-root: ^^ gra1 is back online now thanks to amorin, probably worth watching things a bit later today | 11:50 |
| *** yboaron_ has joined #openstack-infra | 11:58 | |
| *** cmurphy is now known as cmorpheus | 12:05 | |
| *** dkehn has quit IRC | 12:10 | |
| *** smarcet has joined #openstack-infra | 12:10 | |
| openstackgerrit | Andriy Shevchenko proposed openstack/ansible-role-cloud-launcher master: Update home-page https://review.openstack.org/630185 | 12:11 |
| *** kjackal_v2 has quit IRC | 12:13 | |
| *** kjackal has joined #openstack-infra | 12:13 | |
| *** d0ugal has quit IRC | 12:15 | |
| *** smarcet has quit IRC | 12:25 | |
| *** smarcet has joined #openstack-infra | 12:29 | |
| fungi | jiapei: there is a spec open for documenting an updated solution for third-party ci operators, it just lacks someone with third-party ci experience volunteering to write it: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-3rd-party-ci.html | 12:30 |
| fungi | ssbarnea|rover: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?panelId=27&fullscreen&orgId=1&from=now-2d&to=now doesn't seem to indicate ze08 being especially out of space at any point over the past couple of days | 12:33 |
| fungi | arxcruz|ruck: ^ | 12:33 |
| fungi | looks like that was a tripleo-heat-templates build for change 602248,22 | 12:36 |
| *** yboaron_ has quit IRC | 12:37 | |
| fungi | we did record a DISK_FULL result for that build, though it's the only one i find in the scheduler log over the past couple days | 12:37 |
| fungi | looking back, we got a couple more DISK_FULL results on january 6 and 7 | 12:38 |
| *** jpena is now known as jpena|lunch | 12:38 | |
| fungi | those were also both for openstack/tripleo-heat-templates jobs | 12:39 |
| *** smarcet has quit IRC | 12:39 | |
| *** hwoarang has quit IRC | 12:40 | |
| fungi | on the same change | 12:40 |
| fungi | tripleo-ci-centos-7-scenario001-multinode-oooq-container in all three cases | 12:41 |
| fungi | different patchsets, but same project, same job, same change, different zuul executors | 12:41 |
| fungi | ssbarnea|rover: arxcruz|ruck: my guess is you have something specific to that change triggering this | 12:42 |
| *** rpittau is now known as rpittau|lunch | 12:46 | |
| fungi | oh, wow, you have a bot rechecking an empty change 9 times every day? this seems a questionable choice given concerns that tripleo is already using more job resources than the next several most active projects combined | 12:48 |
| fungi | and not just one change, several on different projects and branches? | 12:52 |
| *** smarcet has joined #openstack-infra | 12:54 | |
| *** smarcet has quit IRC | 12:55 | |
| *** dtantsur is now known as dtantsur|brb | 12:59 | |
| fungi | ahh, i was counting comments wrong, it's 3x/day on 6 different changes | 13:00 |
| *** dims has joined #openstack-infra | 13:01 | |
| *** d0ugal has joined #openstack-infra | 13:02 | |
| fungi | so even though it's an empty change, seems there's something about that combination of change 602248 on openstack/tripleo-heat-templates running tripleo-ci-centos-7-scenario001-multinode-oooq-container (perhaps because it's running on the stable/pike branch?) which seems to cause it to be the only one triggering this DISK_FULL build result | 13:02 |
| fungi | 2019-01-11 02:45:20,122 INFO zuul.ExecutorDiskAccountant: /var/lib/zuul/builds/3f8d1e1f50c446d081b9838b4faeccd3 is using 7319MB (limit=5000) | 13:04 |
| fungi | there you go | 13:04 |
| fungi | that happens during the " | 13:05 |
| fungi | that happens during the "upload-logs : Upload logs to log server" task | 13:05 |
| fungi | ssbarnea|rover: arxcruz|ruck: so it looks like tripleo-ci-centos-7-scenario001-multinode-oooq-container on stable/pike tries to archive more than the 5gb limit we're enforcing | 13:06 |
| *** jcoufal has joined #openstack-infra | 13:06 | |
| *** dims has quit IRC | 13:07 | |
| *** dims has joined #openstack-infra | 13:07 | |
| arxcruz|ruck | fungi: hmmmmm, okay, thanks | 13:08 |
| fungi | so, i guess thanks for helping us confirm the rule we put in place to stop tripleo from filling up our logserver again is actually doing its job ;) | 13:09 |
| arxcruz|ruck | fungi: ha! :D | 13:10 |
| arxcruz|ruck | i'll try to figure out with ssbarnea|rover how to fix it | 13:10 |
| arxcruz|ruck | fungi: so, this basically is when we try to upload logs right ? not related to disk size in the image | 13:10 |
| fungi | arxcruz|ruck: unless you're archiving the images themselves, i don't see how it would be related | 13:11 |
| *** boden has joined #openstack-infra | 13:11 | |
| arxcruz|ruck | ok, i'll check what we are uploading to l.o.o | 13:11 |
| fungi | basically, every build gets a 5gb archiving quota for logs and related artifacts it can copy off the build nodes to the executor, and a DISK_FULL result indicates that limit has been exceeded | 13:12 |
| arxcruz|ruck | ok | 13:12 |
| arxcruz|ruck | thanks | 13:12 |
| fungi | (we thought 5gb was already quite a lot, so it's more of a sanity check really, ideally jobs should copy nowhere near that) | 13:12 |
| amorin | frickler: let me check the quota on gra1 | 13:13 |
| ssbarnea|rover | fungi: wow, i never expected that, from the msg I suspected that the worked run out of space. | 13:13 |
| ssbarnea|rover | and indeed, for me I would cap the build limit to 1GB. 5GB far more than enough. | 13:14 |
| amorin | done | 13:14 |
| ssbarnea|rover | I wonder how to fix the table to include the error on it, as most people would miss to see such failure (the have ci msg toggled off) | 13:15 |
| fungi | ssbarnea|rover: arxcruz|ruck: i'll try to get an accounting of some of the other builds of that same job+patchset and see what it comes in at normally | 13:15 |
| fungi | ssbarnea|rover: probably just needs adjusting in https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/gerrit/hideci.js but i haven't looked closely | 13:16 |
| ssbarnea|rover | fungi: yep, i know where the code is, but i really struggle with JS. had a brief look. | 13:17 |
| fungi | you and me both ;) | 13:18 |
| *** rlandy has joined #openstack-infra | 13:18 | |
| ssbarnea|rover | i wonder who are the key JS people that could help us around. | 13:18 |
| fungi | i suspect horizon has more javascript-savvy folks than any other team in openstack | 13:19 |
| ssbarnea|rover | fungi: this was my next question: which is the best irc channel to ask help with js... | 13:20 |
| fungi | oh, i see part of the problem. because the job was aborted during log collection there's no log url included in the report and so the finger url for the console log stream ends up substituted instead | 13:22 |
| fungi | i think there may be an outstanding todo for zuul to at a minimum still upload the console log when that happens | 13:23 |
| *** aojea has quit IRC | 13:25 | |
| ssbarnea|rover | fungi: regarding the disk_full, i wonder if would not be better to add an extra param to rsync to limit max file size. | 13:25 |
| ssbarnea|rover | may not be ideal but may avoid such accidents | 13:26 |
| fungi | ssbarnea|rover: it's possible that can be added to the synchronize task via an ansible parameter for specifying additional rsync options | 13:26 |
| ssbarnea|rover | yep, still not sure to which one to add it but capping to 1GB sounds like a way to avoid it. | 13:27 |
| fungi | okay, so the builds of that same patchset and job both before and after the DISK_FULL run weigh in at 85M and 86M according to du -sh | 13:28 |
| *** jamesmcarthur has joined #openstack-infra | 13:28 | |
| *** jamesmcarthur has quit IRC | 13:28 | |
| fungi | i expect there's some infrequent failure mode for that job when run on stable/pike which causes some logfile to blow up massively | 13:29 |
| ssbarnea|rover | fungi: was the full disk happening durring upload to logs or during collection from nodes -> zuul worker. | 13:29 |
| fungi | looks like it happens in between. from what i could see zuul.ExecutorDiskAccountant reported during the "Upload logs to log server" task, so i think that means one of the first things the task does is check how much data it's about to upload | 13:30 |
| ssbarnea|rover | i need to know which task failed, in order create a CR to cap the filesize. now i see nothing. | 13:30 |
| fungi | https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/upload-logs/tasks/main.yaml#n26 | 13:31 |
| AJaeger | fungi: could you +2A https://review.openstack.org/629439 once no meetings happen, please? (or go with https://review.openstack.org/626185 since that one has some depends-on on it) | 13:32 |
| fungi | ssbarnea|rover: it seems the zuul executor disk accountant is a separate thread of the executor daemon, not part of the ansible running there https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n79 | 13:33 |
| fungi | i think that thread woke up after the files had been retrieved from the nodes but before or during upload to the logserver, and aborted the job out of band | 13:34 |
| fungi | ssbarnea|rover: part of the challenge of limiting the amount of data copied to the executor is that it pulls files from each node in the nodeset for a given build, and each of those is a separate ansible synchronize call/rsync invocation | 13:36 |
| fungi | AJaeger: thanks for the heads-up on those changes. will do | 13:37 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Avoid zuul DISK_FULL failure with too big logs https://review.openstack.org/630224 | 13:37 |
| fungi | ssbarnea|rover: yeah, looking deeper at what transpired on the executor, i doubt 630224 is going to help since the accountant thread could wake up at any point, see a particular build is consuming more disk on the executor than allowed, and abort the job | 13:39 |
| *** janki has joined #openstack-infra | 13:39 | |
| *** janki has quit IRC | 13:39 | |
| ssbarnea|rover | fungi: i know we cannot have a perfect logic for this but if we introduce capping in those places that are likely to cause problems we should be in much better case, right? | 13:40 |
| ssbarnea|rover | i just need more feedback on other tasks where this needs to be added. | 13:41 |
| fungi | we could try to institute a per-node rsync limit instead during log collection, i suppose, though the total amount copied could still exceed the accountant's threshold if the rsync limit multiplied by the number of nodes in the nodeset is larger | 13:41 |
| ssbarnea|rover | i think that in 99/100 cases we have one huge log causing the issue, one that becaume like this due to an endless loop kind of bug. so capping single file size to ~1/5 of total, should be a good enough approach. | 13:42 |
| fungi | looking at the particular case you experienced at least, that job normally only generates ~85mb worth of logs, but occasionally generates more like 7+gb worth of logs instead. limiting how much of that we'll copy to the logserver doesn't prevent the executor's disk accountant thread from aborting the job before it even uploads the logs | 13:42 |
| ssbarnea|rover | later we can improve logging and add some extra truncating before rsync runs | 13:43 |
| fungi | the only way to really impact this, i think, is to limit the data we retrieve from the nodes in the first place so it doesn't wind up on the executor | 13:43 |
| *** jpena|lunch is now known as jpena | 13:44 | |
| fungi | mordred and pabelanger have also been working on the file copying parts of the stdlib in the past week or so, and may have deeper insights too | 13:44 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Avoid zuul DISK_FULL failure with too big logs https://review.openstack.org/630224 | 13:46 |
| *** rpittau|lunch is now known as rpittau | 13:46 | |
| *** priteau has joined #openstack-infra | 13:47 | |
| ssbarnea|rover | i added all those mentioned to the change in order to get extra feedback. | 13:47 |
| openstackgerrit | Will Szumski proposed openstack-dev/pbr master: Do not globally replace path prefix https://review.openstack.org/629006 | 13:54 |
| fungi | ssbarnea|rover: since that's a change to the zuul-jobs standard library, we probably ought to move further discussion out of #openstack-infra to #zuul or the zuul-discuss ml (or comments on the proposed change, of course) | 13:55 |
| *** dkehn has joined #openstack-infra | 13:55 | |
| *** jamesmcarthur has joined #openstack-infra | 13:56 | |
| *** shardy has joined #openstack-infra | 13:57 | |
| *** lbragstad has joined #openstack-infra | 13:57 | |
| *** jamesmcarthur has quit IRC | 14:01 | |
| *** hwoarang has joined #openstack-infra | 14:04 | |
| *** dave-mccowan has joined #openstack-infra | 14:08 | |
| *** sthussey has joined #openstack-infra | 14:08 | |
| *** jtomasek has joined #openstack-infra | 14:10 | |
| *** kgiusti has joined #openstack-infra | 14:22 | |
| *** kjackal has quit IRC | 14:23 | |
| *** kjackal_v2 has joined #openstack-infra | 14:23 | |
| *** weshay has joined #openstack-infra | 14:26 | |
| *** ykarel is now known as ykarelaway | 14:27 | |
| openstackgerrit | Will Szumski proposed openstack-dev/pbr master: Fix white space handling in file names https://review.openstack.org/629161 | 14:31 |
| *** whoami-rajat has quit IRC | 14:36 | |
| *** edmondsw has quit IRC | 14:40 | |
| pabelanger | ianw: neat, the switch to fedora-29 didn't break testing of windmill. That is a first :) | 14:44 |
| *** stephenfin is now known as finucannot | 14:45 | |
| *** kopecmartin is now known as kopecmartin|off | 14:45 | |
| *** jamesmcarthur has joined #openstack-infra | 14:48 | |
| *** ykarelaway has quit IRC | 14:54 | |
| *** whoami-rajat has joined #openstack-infra | 14:55 | |
| *** bobh has joined #openstack-infra | 14:58 | |
| *** rpittau has quit IRC | 15:03 | |
| *** rpittau has joined #openstack-infra | 15:03 | |
| *** dtantsur|brb is now known as dtantsur | 15:06 | |
| *** edmondsw has joined #openstack-infra | 15:08 | |
| *** wolverineav has joined #openstack-infra | 15:10 | |
| *** lpetrut has quit IRC | 15:10 | |
| *** jamesmcarthur has quit IRC | 15:13 | |
| *** smarcet has joined #openstack-infra | 15:13 | |
| *** wolverineav has quit IRC | 15:14 | |
| *** rpittau has quit IRC | 15:15 | |
| *** d0ugal has quit IRC | 15:16 | |
| *** rpittau has joined #openstack-infra | 15:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Exclude pipeline config items for openstack-dev/ci-sandbox https://review.openstack.org/630286 | 15:19 |
| arxcruz|ruck | fungi: any chance you have the logs so we can check which files are consuming so much space? | 15:21 |
| ssbarnea|rover | fungi: knowing which file was the cause of the issue could help us implement some extra protections. | 15:22 |
| *** quiquell is now known as quiquell|off | 15:22 | |
| fungi | arxcruz|ruck: not as far as i'm aware. once the executor disk accounting thread noticed how much data had been pulled from the job nodes it aborted the job on the spot and removed the build dir | 15:22 |
| arxcruz|ruck | fungi: i see... thanks | 15:22 |
| *** jamesmcarthur has joined #openstack-infra | 15:23 | |
| *** smarcet has quit IRC | 15:24 | |
| fungi | i'm digging in the executor debug log to see whether we recorded any additional information about it | 15:24 |
| fungi | but there's ~2k lines of debug logging for that one build, so will take some time to reciew | 15:24 |
| fungi | er, review | 15:24 |
| fungi | we set no_log=True on the "Copy files from /home/zuul/workspace/ on node" task owing to how much it spams the logs otherwise | 15:25 |
| *** bobh has quit IRC | 15:27 | |
| *** smarcet has joined #openstack-infra | 15:28 | |
| *** rpittau has quit IRC | 15:29 | |
| *** rpittau has joined #openstack-infra | 15:30 | |
| *** ykarel has joined #openstack-infra | 15:31 | |
| fungi | yeah, i'm not finding much relevant in the executor log to provide any clues as to what file(s) could be the culprit | 15:34 |
| fungi | perhaps toward the end of your job you could analyze the contents of the log tree, record the sizes of some larger files, and then remove or truncate any which are really excessive so that they don't get copied to the executor? | 15:36 |
| fungi | preferably within a routine which runs even if the job itself fails | 15:37 |
| fungi | the problem with analyzing them once they're on the executor is that analysis could be cut short by the disk accountant aborting the job | 15:38 |
| mordred | fungi: maybe we run a remote du, check to see if the total remote size is above $configurable_threshold, and if so, don't copy any remote logs back and instead copy back a file that contains a recursive du report | 15:40 |
| openstackgerrit | Graham Whaley proposed openstack-infra/openstack-zuul-jobs master: Add dco-signoff check files for Kata Containers https://review.openstack.org/630294 | 15:40 |
| mordred | so that if the copying process detects that it would probably trip the disk accounting, it can report "this job produced WAY too much data - and here's the list of what did it" | 15:41 |
| *** jamesmcarthur has quit IRC | 15:41 | |
| mordred | (so similar to ssbarnea|rover's patch, but wouldn't be targetting specific file sizes and instead would be looking at the aggregate size of zuul-output dir | 15:42 |
| mordred | (obviously, this is depending on finishing landing the zuul-output stack, but it's very close to being done) | 15:42 |
| mordred | fungi: speaking of - https://review.openstack.org/#/c/629571/ | 15:45 |
| *** ekultails has joined #openstack-infra | 15:46 | |
| mordred | fungi: and https://review.openstack.org/#/c/629584 - which I have just rechecked | 15:46 |
| fungi | hrm, yeah a remote du task could be just the ticket | 15:47 |
| *** jistr is now known as jistr|mtg | 15:47 | |
| *** trown|outtypewww is now known as trown|lunch | 15:53 | |
| *** jamesmcarthur has joined #openstack-infra | 15:55 | |
| *** d0ugal has joined #openstack-infra | 15:56 | |
| *** kjackal_v2 has quit IRC | 15:56 | |
| *** kjackal has joined #openstack-infra | 15:56 | |
| ssbarnea|rover | mordred: afaik there is no way to tell rsync to limits how much to collect, so the only practical measure I see is to put a cap on max filesize. | 15:57 |
| ssbarnea|rover | i didn't see any case where we reached max size because multiple-too-big-files, it was always one specific file that was far above decent size. | 15:58 |
| *** markvoelker has joined #openstack-infra | 15:58 | |
| mordred | ssbarnea|rover: we've hit the aggregate max in the past - but this is why I'm suggesting making it a separate task that runs before the rsync | 15:58 |
| *** jistr|mtg is now known as jistr | 15:58 | |
| ssbarnea|rover | mordred: i am not against it, just that i didn't had too much time to work on it. | 15:59 |
| mordred | ssbarnea|rover: totally - it'll take a little bit to get my suggestion written, tested and solid | 16:00 |
| ssbarnea|rover | yep, doing a dry run and triggering a warning could be useful. | 16:00 |
| *** e0ne has quit IRC | 16:00 | |
| ssbarnea|rover | i tried to think about worst case: it will happen again, in random places, with no knowledge about which log would be too big. I wanted to collect something instead of nothing so the developer could be able to do know where to look. | 16:01 |
| ssbarnea|rover | an even better approach would be to truncate big files before collecting them. | 16:02 |
| *** priteau has quit IRC | 16:02 | |
| *** bnemec is now known as beekneemech | 16:02 | |
| ssbarnea|rover | rsync has no support for partial sync, but we could run a task before it, to make logs smaller, keeping only first XXmb from each file (I can assume the repeating-msg/loop is at the end of the file) | 16:03 |
| clarkb | frickler: amorin: thank you for getting gra1 back online | 16:05 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:05 | |
| *** e0ne has joined #openstack-infra | 16:08 | |
| *** tpsilva has joined #openstack-infra | 16:09 | |
| *** jamesmcarthur has quit IRC | 16:09 | |
| *** evrardjp has quit IRC | 16:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add dco-license job https://review.openstack.org/630302 | 16:11 |
| *** evrardjp has joined #openstack-infra | 16:11 | |
| *** dpawlik has quit IRC | 16:12 | |
| *** ramishra has quit IRC | 16:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use promote pipeline for system-config images https://review.openstack.org/629957 | 16:22 |
| *** e0ne has quit IRC | 16:24 | |
| *** e0ne has joined #openstack-infra | 16:24 | |
| *** adriancz has quit IRC | 16:25 | |
| *** EmilienM is now known as EvilienM | 16:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add dco-license job https://review.openstack.org/630302 | 16:30 |
| *** d0ugal has quit IRC | 16:36 | |
| *** pcaruana has quit IRC | 16:36 | |
| clarkb | I guess since it isn't a filesystem limit we don't cause the process to stop running when the limit is hit. If we could manage ti that way then we'd have a chance of copying logs after the earlier copies fail | 16:37 |
| clarkb | but in this case it is an external process that checks jobs and stops the job entirely if they go over | 16:37 |
| clarkb | if only we could tell everyone to run zuul on top of zfs >_> | 16:38 |
| *** e0ne has quit IRC | 16:39 | |
| *** shardy has quit IRC | 16:40 | |
| clarkb | as a heads up I'm going to be in and out today with a variety of tasks like airport taxi and talking to contractors about wall | 16:43 |
| *** ccamacho has quit IRC | 16:44 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/system-config master: Remove 'openstack-gantt' from statusbot monitoring https://review.openstack.org/630323 | 16:44 |
| *** bobh has joined #openstack-infra | 16:44 | |
| anteaya | clarkb: may the construction and taxi'ing go well | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Retire gantt, python-ganttclient https://review.openstack.org/630162 | 16:44 |
| clarkb | AJaeger: I tried to catch up on projet config reviews yesterday. There were a couple that you hadn't reviewed yet if you want to take a quick look too | 16:45 |
| *** iurygregory has quit IRC | 16:45 | |
| *** gyee has joined #openstack-infra | 16:47 | |
| clarkb | anteaya: tahnks | 16:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use a mix-in job to reduce duplication https://review.openstack.org/629983 | 16:51 |
| corvus | clarkb, fungi, mordred: do you have thoughts on multiple inheritance? https://review.openstack.org/629983 | 16:51 |
| *** imacdonn has quit IRC | 16:51 | |
| *** imacdonn has joined #openstack-infra | 16:52 | |
| *** efried is now known as fried_rice | 16:52 | |
| *** rpittau has quit IRC | 16:52 | |
| fungi | it can certainly make things harder to reason about in some circumstances, but i'm not against the concept as a whole | 16:53 |
| fungi | corvus: in this case it reduces duplication, but adds more lines than it removes. though i think i'm in favor because it drops the yaml macros which, on balance, are even more complexity still | 16:55 |
| *** ykarel is now known as ykarel|away | 16:55 | |
| *** tpsilva has left #openstack-infra | 16:56 | |
| corvus | fungi: yeah, i'm wondering if we should jump into the deep end and make 'parent' a list (that should reduce the line count, and maybe make things more clear). or should we just back away slowly and pretend the pool isn't there. :) | 16:56 |
| corvus | i made a picture: http://paste.openstack.org/show/741807/ | 16:56 |
| fungi | what's friday without some fun diagrams, right? | 16:57 |
| fungi | i guess the flow there is top to bottom? it could use some arrows | 16:58 |
| clarkb | I worry that we've already got a complex system that we often don't understand (thats the !corvus set of we) so verbosity for clarity is probably ok? | 16:58 |
| corvus | fungi: well, actually, sort of bottom to top if you read the inheritance path. | 16:59 |
| *** wolverineav has joined #openstack-infra | 17:00 | |
| fungi | sure, but it's not side-to-side or looping around ;) | 17:01 |
| corvus | fungi: it is not pac-man-style inheritance! | 17:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Sketch of explicit multiple-inheritance https://review.openstack.org/630337 | 17:02 |
| corvus | clarkb, fungi: then i wonder if we supported it more explicitly whether it would be easier to follow; that's what that would look like ^ | 17:03 |
| fungi | thanks, easier to see as an example (granted all the zuul inline comments do make it harder to follow in webui/gertty) | 17:04 |
| fungi | so this is basically "what if zuul could take a parent list?" | 17:05 |
| *** wolverineav has quit IRC | 17:05 | |
| corvus | yep | 17:05 |
| *** bobh has quit IRC | 17:05 | |
| fungi | main thing which complicates it from a troubleshooting perspective is precedence, i guess | 17:05 |
| fungi | but we already have that problem anyway, just with a less concise syntax | 17:06 |
| corvus | yeah, though we can choose not to use MI :) | 17:06 |
| *** armax has joined #openstack-infra | 17:06 | |
| corvus | (though it's always there, and it's easy to use accidentally -- any time you add a job and a template with the same job to a project, there you have it :) | 17:07 |
| fungi | either way, i think for people who do want to use them (we might be some of those people too) it does make for an easier read | 17:07 |
| corvus | you can see the inheritance path here: http://logs.openstack.org/83/629983/2/check/system-config-build-image-gitea/b5f477e/zuul-info/inventory.yaml | 17:07 |
| *** jpich has quit IRC | 17:08 | |
| corvus | #118 and #123 are the 2 variants; #449 is the project-pipeline invocation. | 17:08 |
| *** jpich has joined #openstack-infra | 17:08 | |
| fungi | sure, and from a precedence standpoint values in #2 overwrite matching values in #1, #3 overwrites #2, and so on? | 17:08 |
| fungi | er, overwrite values for matching variables i mean | 17:09 |
| openstackgerrit | Andriy Shevchenko proposed openstack/os-performance-tools master: Update home-page https://review.openstack.org/630342 | 17:09 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 17:09 |
| fungi | parameters, arguments, whatever word we like | 17:09 |
| corvus | fungi: yeah. if you look at the inheritance path in the link i just pasted, those are evaluated in order. so for a simple attr like 'timeout', the latest item in the list wins. | 17:10 |
| fungi | perfect, that's exactly what i was asking, yes | 17:10 |
| fungi | i know in some cases we likely merge sets of things as well | 17:11 |
| corvus | fungi: (for pre/post playbooks, that's the onion order -- so read down for pre, and up for post) | 17:11 |
| *** jpich has quit IRC | 17:11 | |
| fungi | got it | 17:11 |
| corvus | and yeah, vars are merged, so that's the order of 'merging' (whatever 'merge' means for each particular data type. sometimes it means overwrite) | 17:11 |
| *** kjackal has quit IRC | 17:13 | |
| *** ginopc has quit IRC | 17:14 | |
| *** armax has quit IRC | 17:18 | |
| *** martinkennelly has joined #openstack-infra | 17:18 | |
| martinkennelly | Hey all, Im using tox to run tests but i want to convert the result to html format. How are you all doing this? | 17:19 |
| martinkennelly | Is there a python script I can pass the output from the test to? | 17:19 |
| clarkb | martinkennelly: we feed the subunit log output from our unittest run through a conversion program that writes html | 17:20 |
| clarkb | let me see if I can find where that script lives | 17:20 |
| martinkennelly | Thank you clark. | 17:21 |
| clarkb | martinkennelly: https://git.openstack.org/cgit/openstack/os-testr/tree/os_testr/subunit2html.py | 17:21 |
| martinkennelly | and the output from tox is passed to that? :) | 17:21 |
| fungi | it's not the output from tox, strictly speaking, it's the subunit data generated by the test runner | 17:21 |
| fungi | or at least recorded by the test runner | 17:22 |
| corvus | so this works if a project uses a test runner that emits subunit output (testr, or stestr for example) | 17:22 |
| fungi | we have a few things we htmlize... are you referring to stuff like http://logs.openstack.org/00/629800/1/gate/openstack-tox-py36/4f2e322/testr_results.html.gz ? | 17:23 |
| corvus | (tox runs stestr) | 17:23 |
| corvus | (for most of our projects) | 17:23 |
| *** Dobroslaw has quit IRC | 17:24 | |
| *** panda is now known as panda|off | 17:25 | |
| martinkennelly | fungi: Exactly this. We ran tox and piped the output to subunit2pyunit | 17:27 |
| fungi | yeah, then that's what we're using subunit2html for | 17:27 |
| fungi | so the thing clarkb linked basically | 17:27 |
| *** armax has joined #openstack-infra | 17:28 | |
| fungi | we also htmlize the stdout/stderr from tox, but we do that in a different way | 17:28 |
| *** trown|lunch is now known as trown | 17:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use promote pipeline for system-config images https://review.openstack.org/629957 | 17:28 |
| martinkennelly | thank you, our test aborted due to a timeout and we saw no logs, we do not know if it killed the output or not :) | 17:28 |
| fungi | in the case of a timeout, stuff like converting subunit files is likely to end up skipped | 17:29 |
| fungi | unless it's performed in a post phase playbook | 17:29 |
| fungi | which gets its own separate timeout from the pre/run timeout | 17:30 |
| *** jtomasek has quit IRC | 17:30 | |
| fungi | (assuming we're talking about a zuul v3 ci system) | 17:30 |
| *** electrofelix has quit IRC | 17:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 17:32 |
| *** lbragstad has quit IRC | 17:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add build-image-jinja-init to check https://review.openstack.org/630351 | 17:35 |
| *** rlandy is now known as rlandy|brb | 17:36 | |
| *** tosky has quit IRC | 17:40 | |
| fungi | clarkb: what do you think is the best way to continue debugging the storyboard-dev replacement server openid errors? authenticate to both the old and new servers and diff the parameters in the redirect url? | 17:43 |
| *** martinkennelly has quit IRC | 17:43 | |
| clarkb | fungi: ya though starting with the new server and just looking at the delta between the urls with the different requests might show something itself? Reading the rfc my hunch is that the url initially sent does not match the one at the end so we fail to verify | 17:44 |
| fungi | before i go too deep, i probably ought to compare a pip freeze too and make sure things like oauth module version are the same and we're not dealing with a regression | 17:45 |
| clarkb | ++ | 17:45 |
| *** dtantsur is now known as dtantsur|afk | 17:48 | |
| *** hwoarang has quit IRC | 17:52 | |
| *** gfidente has quit IRC | 17:58 | |
| *** hwoarang has joined #openstack-infra | 17:59 | |
| *** jtomasek has joined #openstack-infra | 17:59 | |
| *** derekh has quit IRC | 18:01 | |
| *** rlandy|brb is now known as rlandy | 18:07 | |
| *** jtomasek has quit IRC | 18:08 | |
| *** jtomasek_ has joined #openstack-infra | 18:08 | |
| *** TheJulia is now known as needssleep | 18:10 | |
| *** eernst has joined #openstack-infra | 18:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update docs since zuul-web requires zookeeper https://review.openstack.org/630365 | 18:12 |
| *** jtomasek_ has quit IRC | 18:15 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add validate-dco-license role https://review.openstack.org/629565 | 18:16 |
| AJaeger | clarkb: tahnks - will do. | 18:17 |
| AJaeger | fungi, clarkb: could you +2A https://review.openstack.org/629439 once no meetings happen, please? (or go with https://review.openstack.org/626185 since that one has some depends-on on it) - and similar with https://review.openstack.org/630323 | 18:18 |
| *** jamesmcarthur_ has quit IRC | 18:18 | |
| fungi | AJaeger: yep, i starred them earlier when you asked, just waiting until i expect it's safe | 18:18 |
| *** wolverineav has joined #openstack-infra | 18:18 | |
| AJaeger | fungi: ah, thanks | 18:18 |
| AJaeger | fungi: 630323 is new - needs a second +2 as well | 18:19 |
| *** wolverineav has quit IRC | 18:19 | |
| *** wolverineav has joined #openstack-infra | 18:19 | |
| *** psachin has quit IRC | 18:19 | |
| fungi | ahh | 18:20 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add dco-license job https://review.openstack.org/630302 | 18:20 |
| *** ykarel|away has quit IRC | 18:21 | |
| *** remi_ness has joined #openstack-infra | 18:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Simplify dco-license job playbook https://review.openstack.org/630369 | 18:25 |
| *** rkukura has joined #openstack-infra | 18:33 | |
| *** kjackal has joined #openstack-infra | 18:34 | |
| *** smarcet has quit IRC | 18:35 | |
| clarkb | logan- isn't here, but the limestone networking errors have not returned. At this point I think we can probably call that one fixed? | 18:35 |
| fungi | seems so. i guss it was just a startup race on a handful of hypervisor hosts? | 18:37 |
| fungi | the ovs agent started too late or something? | 18:38 |
| clarkb | started too early and interfaces weren't there | 18:38 |
| clarkb | s/ovs/neutron l2 agent/ | 18:38 |
| *** wolverineav has quit IRC | 18:40 | |
| fungi | aha | 18:40 |
| fungi | glad that's known/solved then | 18:40 |
| clarkb | ssbarnea|rover: ^ fyi that is http://status.openstack.org/elastic-recheck/#1810054 maybe we wait for it to fall off completely then mark it fixed? | 18:41 |
| ssbarnea|rover | clarkb: yep, sounds like a good approach. | 18:42 |
| ssbarnea|rover | clarkb: btw, is the 10 days e-r timeframe limited by logstash storage limits? sometimes is usefuly to know how things changed in a bigger timeframe. | 18:44 |
| *** jpena is now known as jpena|off | 18:44 | |
| fungi | it's limited by elasticsearch cluster size/performance | 18:45 |
| clarkb | ssbarnea|rover: yes, we have 1TB drives attached to each elasticsearch cluster node giving us 6TB total storage but with one replica we have 5TB useable | 18:45 |
| clarkb | the other factor is memory constraints related to the on disk storage (elasticsearch aggressively loads data into memory too) | 18:45 |
| fungi | that's currently a 6-way cluster of 60gb ram server instances churning to handle it too, right? | 18:45 |
| ssbarnea|rover | yep, but we could store results in influx db and requery only the last day. obviously it needs man-hours | 18:45 |
| clarkb | fungi: yup | 18:45 |
| clarkb | ssbarnea|rover: we already store raw results in graphite | 18:46 |
| clarkb | and that retention is much longer | 18:46 |
| fungi | also in the zuul sql reporter db | 18:46 |
| fungi | accessible via the builds api | 18:46 |
| *** dmellado has quit IRC | 18:51 | |
| *** gouthamr has quit IRC | 18:51 | |
| *** stevebaker has quit IRC | 18:51 | |
| *** rkukura has quit IRC | 18:52 | |
| *** armax has quit IRC | 18:52 | |
| *** wolverineav has joined #openstack-infra | 18:54 | |
| *** diablo_rojo has joined #openstack-infra | 18:54 | |
| *** kjackal has quit IRC | 18:57 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove 'openstack-gantt' from statusbot monitoring https://review.openstack.org/630323 | 19:06 |
| *** wolverineav has quit IRC | 19:10 | |
| *** wolverineav has joined #openstack-infra | 19:10 | |
| amorin | clarkb: let me know about gra1 if you still have issues | 19:13 |
| clarkb | amorin: will do. So far I haven't noticed anything being unhappy | 19:13 |
| *** wolverineav has quit IRC | 19:14 | |
| *** armax has joined #openstack-infra | 19:24 | |
| *** fried_rice has quit IRC | 19:24 | |
| *** efried has joined #openstack-infra | 19:25 | |
| *** efried is now known as fried_rice | 19:25 | |
| *** rkukura has joined #openstack-infra | 19:26 | |
| *** wolverineav has joined #openstack-infra | 19:29 | |
| *** wolverineav has quit IRC | 19:32 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove meeting-5 and meeting-cp from meetbot https://review.openstack.org/629439 | 19:32 |
| *** wolverineav has joined #openstack-infra | 19:32 | |
| *** ekultails has quit IRC | 19:39 | |
| *** openstackstatus has quit IRC | 19:43 | |
| *** openstackstatus has joined #openstack-infra | 19:43 | |
| *** ChanServ sets mode: +v openstackstatus | 19:43 | |
| *** roman_g has quit IRC | 19:45 | |
| *** dpawlik has joined #openstack-infra | 19:48 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update doc for ptgbot to match current behavior https://review.openstack.org/626869 | 19:49 |
| *** kmalloc has joined #openstack-infra | 19:49 | |
| *** kmalloc has quit IRC | 19:51 | |
| *** gouthamr has joined #openstack-infra | 19:53 | |
| *** dmellado has joined #openstack-infra | 19:58 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Simplify dco-license job playbook https://review.openstack.org/630369 | 20:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Update docs since zuul-web requires zookeeper https://review.openstack.org/630365 | 20:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix docker image builds https://review.openstack.org/630351 | 20:06 |
| *** harlowja has quit IRC | 20:08 | |
| *** eernst has quit IRC | 20:12 | |
| *** armax has quit IRC | 20:14 | |
| *** openstack has joined #openstack-infra | 20:15 | |
| *** ChanServ sets mode: +o openstack | 20:15 | |
| *** harlowja has joined #openstack-infra | 20:18 | |
| *** armax has joined #openstack-infra | 20:19 | |
| *** whoami-rajat has quit IRC | 20:26 | |
| *** e0ne has joined #openstack-infra | 20:26 | |
| *** jamesmcarthur has joined #openstack-infra | 20:28 | |
| *** jamesmcarthur has quit IRC | 20:36 | |
| *** jamesmcarthur has joined #openstack-infra | 20:37 | |
| *** sreejithp has joined #openstack-infra | 20:39 | |
| *** sreejithp has quit IRC | 20:39 | |
| *** sreejithp has joined #openstack-infra | 20:41 | |
| *** remi_ness has quit IRC | 20:43 | |
| fungi | heading out to grab a bite to eat, back in a while | 20:44 |
| anteaya | enjoy food | 20:44 |
| *** jamesmcarthur has quit IRC | 20:46 | |
| *** jamesmcarthur has joined #openstack-infra | 20:47 | |
| *** sreejithp has left #openstack-infra | 20:49 | |
| clarkb | infra-root I've pushed up a stack of changes to openstack-infra/gerrit on a new openstack/2.13.12 branch to start working on support for that update | 20:49 |
| clarkb | https://review.openstack.org/#/c/630390/ is the end of tha tstack | 20:49 |
| corvus | clarkb: thank you thank you thank you | 20:49 |
| clarkb | this gets us a bunch of jgit fixes including a security fix that I don't think really concerns us (you can fetch refs if you know their object names) | 20:50 |
| *** sreejithp has joined #openstack-infra | 20:50 | |
| corvus | clarkb: why is bindep no longer in pre? | 20:50 |
| clarkb | corvus: I don't know, this is a rebase of the existing 2.13 content onto 2.13.12. I removed one change we had cherrypicked from upstream that has since been superceeded by newer upstream work | 20:51 |
| mordred | clarkb: sweet | 20:51 |
| mordred | clarkb: I also have https://review.openstack.org/#/c/610395/ up but I haven't had time to follow up on it yet | 20:51 |
| mordred | clarkb: I couldn't get the buck build to work on the 2.14 branch at all - but the bazel build in 2.15 works ... is the buck build working for you in 2.13? | 20:52 |
| clarkb | corvus: looks like pabelanger is the original author of that, maybe pabelager remembers | 20:52 |
| clarkb | mordred: it was when we switched the jobs over iirc | 20:53 |
| clarkb | mordred: I haven't tried building anything locally yet, I've just done the rebase basically | 20:53 |
| corvus | clarkb: ok. maybe i don't care right now as long as the jobs work :) | 20:53 |
| *** dpawlik has quit IRC | 20:53 | |
| corvus | clarkb: but looking over the list... did the file_id stuff ever get upstreamed, or what's the story with that? | 20:53 |
| clarkb | mordred: also apparently you can upgrade from 2.13 to 2.15 directly now | 20:53 |
| corvus | can we do that? can we do that? /me hops up and down | 20:54 |
| mordred | clarkb: oh great! | 20:54 |
| clarkb | corvus: I don't think mordred got that upstreamed. Story on that was that db broke on mysql because of unsupported indexes | 20:54 |
| clarkb | corvus: mordred reworked the index stuff to support mysql | 20:54 |
| mordred | becuase I'd been thinking of upgrading to 2.15 by upgrading to 2.14 using an upstream build of 2.14 just for the migrations, then an immediate upgrade to our build of 2.15 | 20:54 |
| clarkb | (you can kind of tell that upstream doesn't really run gerrit can't you?) | 20:54 |
| clarkb | mordred: ya it looks like they fixed the migration stuff that was requiring that previously | 20:55 |
| mordred | awesome | 20:55 |
| clarkb | mordred: and it also looks like the sql db to notedb stuff can be done online in the background | 20:55 |
| mordred | SWEET | 20:55 |
| clarkb | so this upgrade has gotten a lot simpler since we first looked at it I think | 20:55 |
| mordred | so upgrading to 2.15 might not be super terrible | 20:55 |
| mordred | yeah | 20:55 |
| *** martinkennelly has joined #openstack-infra | 20:55 | |
| corvus | so do we need a patched 2.15 with file_id in order to perform the upgrade, then we can revert to upstream? | 20:56 |
| clarkb | corvus: that I don't know. Mordred? | 20:56 |
| corvus | https://review.openstack.org/630385 is the change i'm looking at | 20:56 |
| mordred | I thnik I looked at 2.15 and that patch is no longer relevant | 20:57 |
| clarkb | corvus: yup the issue is indexes can only be done on columns 768 bytes wide iirc and filename is 4096 bytes wide (or maybe more) | 20:57 |
| mordred | and that we could basically use upstream ... EXCEPT - if we build our own there's actually a nice story with plugins in 2.15 | 20:57 |
| clarkb | so we added file_id and index on that as unique identifier, it should mostly be safe as long as they don't rely on relative order of fields to move data around? | 20:57 |
| mordred | in that we can do all the plugin inclusion at build time - and our install config management doesn't need to care about it anymore | 20:58 |
| clarkb | mordred: oh that is nice | 20:58 |
| corvus | mordred: so we don't need that patch even temprorarily while the sql->note happens? | 20:58 |
| mordred | yeah. that's what I've got going in that patch - it does the build with our plugins included | 20:58 |
| mordred | so there's just a single war | 20:58 |
| clarkb | er sorry primary key not index (though primary key is an implied index) | 20:58 |
| mordred | corvus: no - because that patch is just a patch to a sql migration - so our db will continue to have the index | 20:59 |
| corvus | mordred: oh, derp thx | 20:59 |
| corvus | we pushed a docker image! https://hub.docker.com/r/opendevorg/jinja-init | 20:59 |
| clarkb | hrm no jobs running at all on that branch. What did I miss | 20:59 |
| mordred | corvus: that is exciting! | 20:59 |
| corvus | note https://hub.docker.com/r/opendevorg/jinja-init/tags | 21:00 |
| clarkb | any idea why we seem to be ignoring that new branch? openstack/2.13.12 to be specific | 21:00 |
| clarkb | corvus: neat | 21:00 |
| corvus | clarkb: i see a gerrit-buck job running for 630390 | 21:00 |
| clarkb | rm I had to hard refresh zuul status to see them | 21:01 |
| clarkb | but see them now, sorry for the pebkac | 21:01 |
| clarkb | mordred: ^ it juts passed the buck job if you want to double check it for working buck | 21:01 |
| corvus | we've now uploaded 3 docker images! https://hub.docker.com/u/opendevorg | 21:01 |
| *** corvus is now known as thecount | 21:02 | |
| thecount | ha ha ha! | 21:02 |
| *** thecount is now known as corvus | 21:02 | |
| mordred | corvus: yay!!! | 21:03 |
| corvus | we should know if promote works in about 15 minutes if the puppet stuff passes | 21:03 |
| mordred | corvus: I was just looking to see how one sets a description ... so far all I've found is mentions of dockerhub reading readmes out of github repos | 21:03 |
| *** jamesmcarthur has quit IRC | 21:04 | |
| corvus | mordred: my first thought was "oh i should check the api docs" and then "oh, right, i'd really rather not" | 21:04 |
| clarkb | we don't seem to upload to tarballs/ci/test/ anymore though | 21:04 |
| clarkb | http://logs.openstack.org/90/630390/1/check/gerrit-buck/159a508/gerrit/gerrit/buck-out/gen/gerrit-v2.13.12.8.6c12de7.war they go into the logs | 21:05 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Update review-dev.o.o to puppet 4 https://review.openstack.org/629667 | 21:06 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 21:06 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Test new Gerrit 2.13.12 war on review-dev https://review.openstack.org/630392 | 21:07 |
| clarkb | mordred: corvus ^ that should apply the war to review-dev that was built at the end of that stack in check | 21:07 |
| clarkb | if we want to merge things instead we should get a war in the normal spot on tarballs | 21:07 |
| clarkb | mordred: I think we want to get to ^ then use that as starting point for 2.15 upgrade | 21:08 |
| *** jamesmcarthur has joined #openstack-infra | 21:08 | |
| mordred | clarkb: ++ | 21:10 |
| clarkb | cmorpheus: ^ as an fyi we might be slightly distracted with review-dev gettin gsome uptsream patches in | 21:10 |
| mordred | clarkb: +2 on that patch - let's go ahead and apply it - I don't expect that to be much of a hard upgrade | 21:11 |
| cmorpheus | clarkb: ah do you want me to pick a different guinea pig? | 21:11 |
| clarkb | cmorpheus: that might help move things on that front along quicker | 21:11 |
| cmorpheus | kk | 21:11 |
| clarkb | mordred: ya I agree, biggest changes are in jgit | 21:11 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Update etherpad-dev.o.o to puppet 4 https://review.openstack.org/629667 | 21:13 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 21:13 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix docker image builds https://review.openstack.org/630351 | 21:13 |
| clarkb | mordred: I'm going to have to pop out soon to do a round trip to the airport and back, but if you've got enough bw to help shepherd that in this afternoon that would be great | 21:14 |
| clarkb | otherwise I'll pick it up again after the weekend I guess | 21:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 21:14 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Change openstack-dev to openstack-discuss https://review.openstack.org/625388 | 21:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Also report promote results to gerrit https://review.openstack.org/630393 | 21:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix image promote playbook https://review.openstack.org/630395 | 21:24 |
| corvus | mordred: ^ would you review those 2 please? | 21:25 |
| mordred | corvus: +A on the first - +2 on the second, although i think it's simple enough you could go ahead and +A if you wanted | 21:26 |
| clarkb | I just approved it | 21:27 |
| corvus | \o/ | 21:27 |
| clarkb | corvus: when you get a sec can you review https://review.openstack.org/#/c/630392/1 to update review-dev to 2.13.12? | 21:27 |
| corvus | clarkb: the first 3 gerrit changes don't have zuul jobs, so may be difficult to approve | 21:27 |
| corvus | clarkb: done | 21:28 |
| clarkb | corvus: oh good point, would you prefer I reorder things or we force merge the bottom of the stack? | 21:28 |
| mordred | corvus: you want to +A https://review.openstack.org/#/c/630392/ real quick? | 21:28 |
| mordred | oh - clarkb already said that | 21:28 |
| mordred | :) | 21:28 |
| clarkb | if we reorder stack 630392 might be a little weird (to ensure we test the actual war, but we can update it to point at the merged war later) | 21:28 |
| *** jamesmcarthur has quit IRC | 21:28 | |
| corvus | clarkb: maybe add in a noop job at the top? | 21:28 |
| corvus | clarkb: yeah, i think we should merge the changes; so maybe we should add the noop, merge changes, then point -dev at the merged war | 21:30 |
| clarkb | corvus: ya I'm working on adding noop to the .gitreview change and will make that change the first one | 21:30 |
| mordred | clarkb: if you have a sec, wanna poke https://review.openstack.org/#/c/629571/ adn https://review.openstack.org/#/c/629584/ ? (they're both safe) | 21:30 |
| corvus | clarkb: but we can probably still restart -dev while that's going on. just restart it a second time as a sanity check. | 21:30 |
| *** jamesmcarthur has joined #openstack-infra | 21:30 | |
| clarkb | corvus: ++ | 21:30 |
| mordred | corvus: ++ | 21:31 |
| *** rlandy has quit IRC | 21:31 | |
| clarkb | ok stack updated now with .zuul.yaml in first change running noop jobs | 21:33 |
| *** armax has quit IRC | 21:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Also report promote results to gerrit https://review.openstack.org/630393 | 21:37 |
| *** PrinzElvis has joined #openstack-infra | 21:38 | |
| clarkb | mordred: shouldn't those jobs just copy to the dir that fetch output will fetch from? | 21:38 |
| clarkb | mordred: I guess this is a transitional piece that allows existing jobs to publish like jobs that are updated? | 21:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix image promote playbook https://review.openstack.org/630395 | 21:39 |
| corvus | i'm going to self-reapprove that -- it just adds comments to trigger the builds | 21:39 |
| mordred | clarkb: the jobs should copy to the dir that fetch output will fetch from - but this is to run after fetch-output - so that in openstack's install we'll publish pre-approval artifacts and docs to our log server ... this lets other deployers use the zuul-output dir on remote nodes without being tied to our scheme of publishing stuff to the log server | 21:40 |
| clarkb | gotcha | 21:41 |
| mordred | in theory at least :) | 21:42 |
| *** tosky has joined #openstack-infra | 21:42 | |
| *** jamesmcarthur has quit IRC | 21:43 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Update etherpad-dev.o.o to puppet 4 https://review.openstack.org/629667 | 21:46 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 21:46 |
| clarkb | the gerrit stack is Zuul +1 now | 21:51 |
| clarkb | corvus: mordred ^ | 21:51 |
| *** trown is now known as trown|outtypewww | 21:52 | |
| corvus | clarkb: lgtm. maybe we revisit some of that on the 2.15 branch. :) | 21:54 |
| clarkb | corvus: ya I'm thinking we may be be able to shift it to a job we run with a branch parameter? if we can stop needing to carry patches that is | 21:55 |
| *** jamesmcarthur has joined #openstack-infra | 21:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Increase number of k8s nodes to 4 https://review.openstack.org/628031 | 21:56 |
| corvus | rebase ^ | 21:56 |
| clarkb | fwiw we had shifted to openstack/2.13 to try and make a central branch | 21:57 |
| clarkb | but I think that makes less sense in the context of zuulv3 | 21:57 |
| clarkb | since the jobs aren't externally managed and have to be updated iwth a rebase on top of latest code anyway | 21:57 |
| *** jamesmcarthur has quit IRC | 21:57 | |
| clarkb | made sense with v2.5 though | 21:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Test new Gerrit 2.13.12 war on review-dev https://review.openstack.org/630392 | 22:04 |
| clarkb | woo | 22:04 |
| clarkb | but now I must afk for a bit | 22:05 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add role to move docs and artifacts to log root https://review.openstack.org/629571 | 22:07 |
| *** jcoufal has quit IRC | 22:10 | |
| *** EvilienM is now known as EmilienM | 22:14 | |
| *** rtjure has quit IRC | 22:15 | |
| *** gagehugo has quit IRC | 22:15 | |
| *** gagehugo has joined #openstack-infra | 22:17 | |
| *** e0ne has quit IRC | 22:19 | |
| *** rtjure has joined #openstack-infra | 22:23 | |
| *** armax has joined #openstack-infra | 22:27 | |
| *** sthussey has quit IRC | 22:28 | |
| *** boden has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix image promote playbook https://review.openstack.org/630395 | 22:30 |
| corvus | clarkb, mordred, fungi: it worked! https://hub.docker.com/r/opendevorg/gitea/tags | 22:32 |
| *** jcoufal has joined #openstack-infra | 22:32 | |
| corvus | end-to-end docker image promotion | 22:32 |
| corvus | see the promote report here: https://review.openstack.org/630395 | 22:32 |
| *** markvoelker has quit IRC | 22:32 | |
| clarkb | corvus: it is good they sort in reverse order as I imagine that list will get long | 22:34 |
| corvus | i've rechecked https://review.openstack.org/629931 (which didn't fail before because we forgot to add it to the check pipeline) | 22:34 |
| corvus | clarkb: ayup | 22:34 |
| corvus | oh, er, actually, we don't have a build for gitea-init yet | 22:35 |
| clarkb | how you know traffic is bad in portland, Google maps suggets driving through vancouver washington to get to the airport | 22:36 |
| corvus | wow | 22:36 |
| clarkb | looks like it is only a few minutes worse than the normal route :) | 22:37 |
| fungi | clearly i should disappear to the friday afternoon bar more often; that seems to cause exciting things to happen | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Build a gitea-init image https://review.openstack.org/630404 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use the v1.6.0 template instead of master https://review.openstack.org/629936 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update to gitea v1.6.3 https://review.openstack.org/629942 | 22:38 |
| corvus | gah what is it with me and the check pipeline | 22:40 |
| clarkb | tehre is no check only gate? | 22:41 |
| clarkb | I think next puppet run in a couple minutes will update the review-dev war | 22:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Build a gitea-init image https://review.openstack.org/630404 | 22:41 |
| clarkb | re portland traffic my drive to the airport will be longer than the flight from seattle | 22:41 |
| clarkb | people that live in proper cities are probably loling at me | 22:42 |
| *** e0ne has joined #openstack-infra | 22:45 | |
| fungi | cmorpheus: i appreciate the casual friday nick. even if it's not a sandman reference. it's not a sandman reference is it? | 22:48 |
| corvus | i suddenly want to take a red pill | 22:50 |
| mordred | corvus: zomg. that's awesome | 22:51 |
| *** e0ne has quit IRC | 22:51 | |
| fungi | oh, sure, could be a matrix reference, but simpsons^Wsandman did it first | 22:53 |
| mordred | fungi: it could be first initial c, middle initial m, lastname orpheus and could be a greek reference | 22:54 |
| fungi | well, morpheus and orpheus are both greek mythological references, yes | 22:54 |
| mordred | fungi: good point | 22:55 |
| fungi | orpheus went to the underworld to rescue euridice | 22:55 |
| fungi | morpheus was the god of sleep/dreams | 22:55 |
| fungi | unless my memory is failing me | 22:55 |
| mordred | yah. those are both accurate | 22:55 |
| clarkb | ok I've tried to hold out long enough for puppet to run but I have to drive a car now | 22:56 |
| clarkb | mordred: fungi corvus ^ if puppet runs soonish maybe you can restart gerrit on review-dev? | 22:56 |
| mordred | yah | 22:56 |
| fungi | clarkb: go operate a moving vehicle. we've got this | 22:56 |
| mordred | fungi: you are so optimistic | 22:56 |
| mordred | fungi: I expect complete chaos every time clarkb is away from his keyboard | 22:57 |
| fungi | at least i didn't ask anyone to "hold my beer" | 22:57 |
| mordred | fungi: :) | 22:57 |
| mordred | fungi: the real skill comes in doing stupid things without putting the beer down :) | 22:57 |
| fungi | in hindsight, that probably would have curtailed the spillage | 22:57 |
| corvus | when i think of fungi, i think "that guy can hold his beer". | 22:58 |
| fungi | in multiple ways | 22:58 |
| imacdonn | clarkb: did you ever get any followup from the neutron team re. http://status.openstack.org/elastic-recheck/#1808171 ? | 23:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: run a gerrit container https://review.openstack.org/630406 | 23:04 |
| imacdonn | clarkb: I really should have read the bug report before asking ... doing that now | 23:05 |
| corvus | fungi: https://review.openstack.org/630404 could use a +3 | 23:05 |
| fungi | cool, on it | 23:08 |
| fungi | Jan 11 23:03:18 review-dev01 puppet-user[30401]: (/Stage[main]/Gerrit/Exec[gerrit-initial-index]) Failed to call refresh: Command exceeded timeout | 23:14 |
| fungi | i guess that's what clarkb was waiting to see | 23:14 |
| fungi | assuming all we need now is a `sudo service restart gerrit` | 23:14 |
| *** rascasoft has quit IRC | 23:19 | |
| *** rascasoft has joined #openstack-infra | 23:26 | |
| *** slaweq has quit IRC | 23:28 | |
| fungi | assuming so since it wasn't actually running | 23:31 |
| fungi | Job for gerrit.service failed because the control process exited with error code. See "systemctl status gerrit.service" and "journalctl -xe" for details. | 23:31 |
| fungi | according to the error_log: | 23:32 |
| fungi | No index versions ready; run java -jar /home/gerrit2/review_site/bin/gerrit.war reindex | 23:32 |
| fungi | running that now as gerrit2 with ~gerrit2/review_site as cwd | 23:34 |
| *** rascasoft has quit IRC | 23:37 | |
| *** fried_rice has quit IRC | 23:37 | |
| *** rascasoft has joined #openstack-infra | 23:44 | |
| fungi | this... is gonna take a while | 23:45 |
| *** hwoarang has quit IRC | 23:46 | |
| *** hwoarang has joined #openstack-infra | 23:47 | |
| openstackgerrit | Merged openstack-infra/system-config master: Build a gitea-init image https://review.openstack.org/630404 | 23:48 |
| *** efried has joined #openstack-infra | 23:49 | |
| corvus | another success! https://hub.docker.com/r/opendevorg/gitea-init/tags | 23:51 |
| *** pabelanger has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
