Saturday, 2019-01-12

*** harlowja has quit IRC00:01
clarkbfungi it timed out in puppet then hadno indexes?00:02
clarkbI guess this is why wetest with the dev server00:02
clarkbimacdonn: slaweq was going to try and look at it00:02
clarkbfungi I wonder if the puppet didnt do what we wanted or if the indexes really were gone00:05
imacdonnclarkb: yeah, looks like he did, but has not yet gotten to the bottom of it (based on comment from earlier today)00:05
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Fix noop job toDict()  https://review.openstack.org/63040900:08
fungiReindexed 107659 documents in changes index in 1184.9s (90.9/s)00:11
fungiand now i've started the service00:12
*** jcoufal has quit IRC00:13
fungiseems we've got the webui back now00:13
fungi by Gerrit Code Review (2.13.12-8-g6c12de7)00:14
fungilooks right to me00:14
*** rascasoft has quit IRC00:16
*** gyee has quit IRC00:21
corvusinfra-root, config-core: first thing next week, i'll work on factoring out the docker build/promote stuff into zuul-jobs roles and send emails to the lists telling folks about the pattern.00:22
*** rascasoft has joined #openstack-infra00:22
fungiit's already awesome00:23
corvusfungi: review-dev looks like a ui to me!00:23
fungisoon to be reconsumable awesome, all the better00:23
*** sreejithp has quit IRC00:24
*** tosky has quit IRC00:24
corvusfungi, clarkb: so it's no clear to me what we should do in prod with review00:25
*** rascasoft has quit IRC00:31
*** rascasoft has joined #openstack-infra00:38
fungii think the puppet exec to trigger an offline reindex on upgrade (or even a restart on upgrade) is an issue since it can't complete within the exec timeout00:38
fungibetter to do the upgrade without the restart triggered, and then perform a manual restart during a maintenance window?00:38
*** chandankumar has quit IRC00:46
*** chandankumar has joined #openstack-infra00:47
*** rascasoft has quit IRC00:53
*** chandankumar has quit IRC00:53
*** chandankumar has joined #openstack-infra00:54
*** armax has quit IRC01:05
*** rascasoft has joined #openstack-infra01:08
*** slaweq has joined #openstack-infra01:13
*** armax has joined #openstack-infra01:15
clarkbfungi or extend the timeout? that is what 20 minutes? that seems long but ya01:19
*** chandankumar has quit IRC01:20
*** armax has joined #openstack-infra01:21
clarkbwhat I dont getis we shouldnt need to do an offline reindex01:21
clarkbits the same version if gerrit (mostly)01:21
clarkbI wonderif puppet didnt detect that properly and did the wrong thing01:22
*** chandankumar has joined #openstack-infra01:22
*** slaweq has quit IRC01:25
*** armax has quit IRC01:25
*** rascasoft has quit IRC01:27
*** chandankumar has quit IRC01:30
*** chandankumar has joined #openstack-infra01:32
*** rascasoft has joined #openstack-infra01:35
*** wolverineav has quit IRC01:46
*** wolverineav has joined #openstack-infra01:46
*** wolverineav has quit IRC01:54
*** pabelanger has joined #openstack-infra01:56
*** dave-mccowan has quit IRC02:05
*** rfolco has quit IRC02:07
*** rascasoft has quit IRC02:10
*** jamesmcarthur has joined #openstack-infra02:10
*** slaweq has joined #openstack-infra02:11
*** rascasoft has joined #openstack-infra02:11
*** jamesmcarthur has quit IRC02:14
*** slaweq has quit IRC02:15
*** diablo_rojo has quit IRC02:23
*** rascasoft has quit IRC02:33
*** pabelanger has quit IRC02:36
*** rascasoft has joined #openstack-infra02:37
*** rfolco has joined #openstack-infra02:39
openstackgerritAndriy Shevchenko proposed openstack/gertty master: Update home-page  https://review.openstack.org/63012002:48
*** rascasoft has quit IRC02:48
*** rascasoft has joined #openstack-infra02:53
*** yamamoto has joined #openstack-infra03:03
*** yamamoto_ has joined #openstack-infra03:07
*** yamamoto_ has quit IRC03:08
*** yamamoto_ has joined #openstack-infra03:09
*** yamamoto has quit IRC03:10
*** rascasoft has quit IRC03:10
*** yamamoto_ has quit IRC03:16
*** apetrich has quit IRC03:16
*** anteaya has quit IRC03:16
*** rascasoft has joined #openstack-infra03:19
*** zaneb has joined #openstack-infra03:29
*** rkukura has quit IRC03:41
*** armax has joined #openstack-infra03:53
openstackgerritJames E. Blair proposed openstack-infra/system-config master: WIP: run a gerrit container  https://review.openstack.org/63040604:03
*** zaneb has quit IRC04:05
*** slaweq has joined #openstack-infra04:11
*** slaweq has quit IRC04:16
*** hwoarang has quit IRC04:24
*** hwoarang has joined #openstack-infra04:24
*** rfolco has quit IRC04:27
*** rascasoft has quit IRC04:34
*** rascasoft has joined #openstack-infra04:40
*** weshay has quit IRC05:19
*** rascasoft has quit IRC05:30
*** ykarel|away has joined #openstack-infra05:32
*** rascasoft has joined #openstack-infra05:39
*** martinkennelly has quit IRC05:41
*** rascasoft has quit IRC05:45
*** rascasoft has joined #openstack-infra05:51
*** rascasoft has quit IRC05:56
*** whoami-rajat has joined #openstack-infra06:02
*** rascasoft has joined #openstack-infra06:03
*** slaweq has joined #openstack-infra06:11
*** slaweq has quit IRC06:16
*** rascasoft has quit IRC06:18
*** lpetrut has joined #openstack-infra06:47
*** rascasoft has joined #openstack-infra07:06
*** slaweq has joined #openstack-infra07:11
*** slaweq has quit IRC07:16
*** armax has quit IRC07:18
*** slaweq has joined #openstack-infra07:50
*** lpetrut has quit IRC08:00
*** ykarel|away has quit IRC08:00
*** slaweq has quit IRC08:05
*** irdr has quit IRC08:08
*** jamesmcarthur has joined #openstack-infra08:11
*** jamesmcarthur has quit IRC08:16
*** lpetrut has joined #openstack-infra08:18
*** jbadiapa has quit IRC08:29
*** slaweq has joined #openstack-infra08:29
*** cmorpheus is now known as cmurphy08:56
openstackgerritMerged openstack-infra/project-config master: Add merge-output-to-logs to base-test  https://review.openstack.org/62958408:59
*** lpetrut has quit IRC09:14
*** rascasoft has quit IRC09:47
*** rascasoft has joined #openstack-infra09:48
*** e0ne has joined #openstack-infra09:50
*** e0ne has quit IRC09:53
*** ykarel|away has joined #openstack-infra10:08
openstackgerritTobias Henkel proposed openstack-infra/zuul master: WIP: Match tag items against containing branches  https://review.openstack.org/57855710:11
*** remi_ness has joined #openstack-infra10:41
*** d0ugal has joined #openstack-infra10:41
*** d0ugal has quit IRC10:59
*** jtomasek has joined #openstack-infra11:01
*** d0ugal has joined #openstack-infra11:02
*** jaosorior has quit IRC11:07
*** d0ugal has quit IRC11:12
*** yamamoto has joined #openstack-infra11:14
*** yamamoto has quit IRC11:18
*** tosky has joined #openstack-infra11:38
*** ykarel|away has quit IRC11:41
*** dkehn has quit IRC12:06
*** whoami-rajat has quit IRC12:11
*** martinkennelly has joined #openstack-infra12:56
*** dkehn has joined #openstack-infra13:24
*** evrardjp has quit IRC13:34
*** rfolco has joined #openstack-infra13:47
fungiclarkb: i *think* the puppet manifest just runs an offline reindex any time the warfile changes14:08
fungilooking14:08
fungiyeah, the gerrit-reindex exec has subscribe => File['/home/gerrit2/review_site/bin/gerrit.war']14:10
fungioh, though it's wrapped in a conditional for $offline_reindex which defaults to false in the main gerrit class, so unless we're overriding that, it's an automatic behavior of gerrit init14:13
fungiyeah, we don't override that, so i don't think we actually fired the gerrit-reindex exec14:17
fungiJan 11 22:58:18 review-dev01 puppet-user[30401]: (/Stage[main]/Gerrit/Exec[gerrit-init]) Failed to call refresh: /etc/init.d/gerrit stop; /usr/bin/java -jar /home/gerrit2/review_site/bin/gerrit.war init -d /home/gerrit2/review_site --batch --no-auto-start returned 128 instead of one of [0]14:20
fungioh, i bet that's because before that we have14:21
fungiJan 11 22:58:13 review-dev01 puppet-user[30401]: (/Stage[main]/Gerrit/Exec[gerrit-initial-init]) Triggered 'refresh' from 1 events14:21
fungithough there's also14:22
fungiJan 11 23:03:18 review-dev01 puppet-user[30401]: (/Stage[main]/Gerrit/Exec[gerrit-initial-index]) Failed to call refresh: Command exceeded timeout14:22
fungiwhy are we triggering gerrit-initial-index?14:22
fungirereading the source again14:22
fungitriggered by subscribe => Exec['gerrit-initial-init']14:24
fungiwhich also has subscribe => File['/home/gerrit2/review_site/bin/gerrit.war']14:25
fungiso any time we replace /home/gerrit2/review_site/bin/gerrit.war we trigger gerrit-initial-init which in turn triggers gerrit-initial-index14:25
fungieven though the comment for gerrit-initial-index says "We need to make the initial index for a fresh install." so presumably is not meant to happen on upgrade14:26
fungireusing the gerrit container and leaving this spaghetti behind is looking better and better14:27
*** roman_g has joined #openstack-infra14:32
*** rfolco has quit IRC14:54
*** jtomasek has quit IRC15:00
*** kgiusti has left #openstack-infra15:10
*** yamamoto has joined #openstack-infra15:17
*** yamamoto has quit IRC15:22
openstackgerritJames E. Blair proposed openstack-infra/system-config master: WIP: run a gerrit container  https://review.openstack.org/63040615:31
openstackgerritJeremy Stanley proposed openstack-infra/jeepyb master: Correct blueprint topic links for modern Gerrit  https://review.openstack.org/63046115:37
clarkbfungi :( I was worried about that. I'll have to see if there is an easy fix on monday. I expect its mostly an issue because we had to reindex after every upgrade for a while andnow it is mostly online amd our puppet didnt catch up15:38
fungiwell, and i think we hacked triggers into the manifest in the past without noticing they resulted in reindexing more often than was strictly required15:39
*** rfolco has joined #openstack-infra15:49
*** jamesmcarthur has joined #openstack-infra15:52
openstackgerritJeremy Stanley proposed openstack-infra/project-config master: Add puppetforge publish and add to puppet release  https://review.openstack.org/62757315:53
fungitobias-urdin: ^ sorry for the delay15:53
*** e0ne has joined #openstack-infra15:56
*** tosky has quit IRC15:58
*** e0ne has quit IRC16:05
corvusclarkb, fungi: yeah, that sounds about right; i think that was required in the past16:18
mordredfungi: yah - 'gerrit in container' is gonna be much cleaner I think16:18
mordredcorvus: in the docker-compose you have there - you just going with the upstream image rather than the locally built reason for POC expediency?16:20
mordred(I like the patch!)16:20
corvusmordred: yeah, i just wanted to start sketching out a framework16:21
mordredcool. I think the shape of that sketch is fantastic16:21
openstackgerritJames E. Blair proposed openstack-infra/system-config master: WIP: run a gerrit container  https://review.openstack.org/63040616:23
corvusmaybe someday zuul will agree :)16:23
mordredcorvus: bah16:23
mordredcorvus: I'm gonna brainstorm a little bit about a way to take the gerrit dockerfile patch I've got up but have that patch live in system-config (or somewhere else) and see if we can't just point it at the upstream gerrit git repo using multi-repo job in zuul rather than needing to maintain the fork / branch for each release16:25
corvusmordred: oh you have a gerrit dockerfile patch?  i didn't see that16:25
mordredyeah - you even +2'd it :)16:25
mordredhttps://review.openstack.org/#/c/61039516:26
mordredit's for 2.1516:26
corvusthat was so last year!  :)16:26
mordredI should port in your new docker roles (althoughj I see in scrollback you're going to generalize them into zuul-jobs - so maybe I'll wait for that)16:26
corvus++16:27
mordredbut maybe if we do a job that has required-project: https://gerrit.googlesource.com/gerrit - and a pre-playbook that just copies our dockerfile and does the modifications for the plugins16:27
mordredsomething something16:28
mordredcorvus: in fact, if we went that route, we could get rid of the submodule section here: https://review.openstack.org/#/c/610395/4/.gitmodules16:31
*** martinkennelly has quit IRC16:31
mordredand also add its-base, its-storyboard and javamelody as required-projects16:31
corvusand all supporting depends-on.  i'm digging it.16:32
mordredwe'd just need to add https://gerrit-review.googlesource.com as a source :)16:33
corvusseems a reasonable thing to do16:33
corvusi don't think it has an ssh listener though16:33
mordredno? poop. I mean - we could also just use git driver16:34
corvusthat'll at least get us the projects16:35
corvuswe have support for reporting over http; we can probably add querying to that, and make ssh optional, then we'd only lack events, but depends-on would work.16:35
mordreddo they have http events?16:35
corvusafaik no; we'd have to do gertty-style queries16:36
mordrednod16:36
corvus(which means maintaining local state)16:36
mordredwell- gerrithub ci is doing ci on the gerrit gerrit - so maybe there is an ssh interface that they expose to some people?16:37
mordredor maybe it's just also polling16:38
*** ykarel|away has joined #openstack-infra16:42
*** jamesmcarthur has quit IRC16:43
*** jonher has quit IRC16:44
*** jamesmcarthur has joined #openstack-infra16:46
*** jamesmcarthur has quit IRC16:47
*** ykarel|away has quit IRC16:48
*** ykarel|away has joined #openstack-infra16:48
*** whoami-rajat has joined #openstack-infra16:49
*** imacdonn has quit IRC16:51
*** imacdonn has joined #openstack-infra16:52
*** ykarel|away has quit IRC16:54
corvusmordred: success!  https://review.openstack.org/63040616:59
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a bazel image  https://review.openstack.org/63046517:00
mordredcorvus: neat!17:00
mordredcorvus: testinfra produces a junit.xml?17:01
* mordred hadn't noticed that before17:02
*** kopecmartin|off has quit IRC17:03
*** kopecmartin has joined #openstack-infra17:04
*** ykarel|away has joined #openstack-infra17:31
*** yamamoto has joined #openstack-infra17:39
*** yamamoto has quit IRC17:43
*** yamamoto has joined #openstack-infra17:44
*** armax has joined #openstack-infra17:50
*** ykarel|away has quit IRC17:58
*** rkukura has joined #openstack-infra18:00
*** armax has quit IRC18:16
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a bazel image  https://review.openstack.org/63046518:22
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add support for passing path to docker build  https://review.openstack.org/63046718:22
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a gerrit image  https://review.openstack.org/63046818:22
mordredcorvus: ^^ ok. there's the whole thing with dockerfiles in system-config and referencing gerrit source directly rather than in our fork18:22
mordredcorvus: I didn't do any required-projects things yet - and also we probably want to talk about how to handle major versions with the tag/promote workflow18:24
mordredcorvus: I could see us wanting an opendev/gerrit:2.15 and opendev/gerrit:2.16 for instance18:25
*** yamamoto has quit IRC18:32
clarkbadd a tag triggered job that tags docoer image based on git tag?18:33
clarkbruns alongside the commit by commit promotion pipeline18:34
clarkbif you want to avoid building new images you could lookup the change that matches the tag and update the docoer tag on that ?18:34
*** lpetrut has joined #openstack-infra18:36
*** armax has joined #openstack-infra18:37
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a gerrit image  https://review.openstack.org/63046818:45
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a gerrit image  https://review.openstack.org/63046818:47
mordredclarkb: turns out corvus had already thought of the thign I was thinking about ... and that last patch uses it (you can give a list of tags to promote to)18:48
mordredclarkb: I mostly wanted to make sure that once we're running 2.15 and we go to start building 2.16 we don't wind up overwriting the 2.15 images18:48
mordredbut I think that latest patch will do what we want - and we should be able to happily have builds for 2.15 and 2.16 coexist without borking ourselves18:49
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a gerrit image  https://review.openstack.org/63046818:50
*** e0ne has joined #openstack-infra18:50
mordredcorvus, clarkb: also - I'd like to say how pleasing it is that https://review.openstack.org/#/c/630468/ is much smaller than https://review.openstack.org/#/c/61039518:51
*** whoami-rajat has quit IRC18:58
*** whoami-rajat has joined #openstack-infra19:03
*** yamamoto has joined #openstack-infra19:05
*** roman_g has quit IRC19:07
*** yamamoto has quit IRC19:11
*** jonher has joined #openstack-infra19:11
*** lpetrut has quit IRC19:14
*** efried has quit IRC19:28
*** efried1 has joined #openstack-infra19:28
*** efried1 is now known as efried19:29
*** yamamoto has joined #openstack-infra19:41
*** e0ne has quit IRC19:45
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Revert "docs: Added missing -i on docker-compose up command"  https://review.openstack.org/63047019:45
*** yamamoto has quit IRC19:46
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Gracefully close db connections in tests  https://review.openstack.org/63047119:51
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Enforce sql connections for scheduler and web  https://review.openstack.org/63047219:51
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a gerrit image  https://review.openstack.org/63046820:05
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Put bazel image into gerrit Dockerfile  https://review.openstack.org/63047320:05
mordredcorvus, clarkb: ^^ that last patch of the stack is a sake-of-argument - if we prefer that version I can squash it down20:07
*** jonher has quit IRC20:16
*** jonher_ has joined #openstack-infra20:16
*** jonher_ is now known as jonher20:16
*** yamamoto has joined #openstack-infra20:35
*** yamamoto has quit IRC20:40
*** eernst has joined #openstack-infra20:48
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix documented invocation of docker-compose  https://review.openstack.org/63047420:53
*** hongbin has joined #openstack-infra20:54
*** yamamoto has joined #openstack-infra21:08
*** whoami-rajat has quit IRC21:08
corvusmordred: yeah, i vote drop the bazel image21:12
*** yamamoto has quit IRC21:12
corvusmordred: when you squash, tabs need fixing in 63046821:13
*** remi_ness has quit IRC21:18
corvusmordred: all reviewed21:19
mordredcorvus: silly tabs21:25
*** kopecmartin has quit IRC21:27
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add support for passing path to docker build  https://review.openstack.org/63046721:35
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Build a gerrit image  https://review.openstack.org/63046821:35
mordredcorvus: ^^ fixed tabs and docs - thanks. also, squashed21:36
*** yamamoto has joined #openstack-infra21:42
*** yamamoto has quit IRC21:47
openstackgerritMerged openstack-infra/zuul master: Fix documented invocation of docker-compose  https://review.openstack.org/63047421:51
*** vultaire has joined #openstack-infra21:52
*** yamamoto has joined #openstack-infra21:59
*** eernst has quit IRC22:04
*** jonher_ has joined #openstack-infra22:05
*** jonher has quit IRC22:08
*** jonher_ is now known as jonher22:08
*** slaweq has quit IRC22:14
*** jtomasek has joined #openstack-infra22:41
*** rascasoft has quit IRC23:13
*** rkukura_ has joined #openstack-infra23:14
*** rkukura has quit IRC23:17
*** rkukura_ is now known as rkukura23:17
*** quiquell|off has quit IRC23:20
*** quiquell has joined #openstack-infra23:21
*** quiquell has quit IRC23:22
*** quiquell has joined #openstack-infra23:22
*** quiquell has quit IRC23:23
*** quiquell has joined #openstack-infra23:23
*** mtreinish has quit IRC23:36
*** rascasoft has joined #openstack-infra23:43
*** rascasoft has quit IRC23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!