Friday, 2020-12-18

*** brinzhang0 has quit IRC00:02
openstackgerritMerged openstack/project-config master: nodepool: pause opensuse-15 builds
ianwcopying it via https locally, and via rsync locally it is *very* slow.  wgetting it from another mirror it comes in at ~2mb/s00:13
ianwthis is pointing to an afs issues00:13
ianwi/o to the disk is ok 209715200 bytes (210 MB, 200 MiB) copied, 7.11467 s, 29.5 MB/s00:15
ianwi'm going to reboot it there's a few old afs errors but nothing major00:17
corvusinfra-root: fyi jitsi merged my etherpad patch, so we should be able to use upstram docker containers next time they update.  update frequencies are kinda random though, so may be a while.  just need to check periodically00:18
*** mlavalle has quit IRC00:32
ianwi'm not really sure what to do.  the airship mirror gets the big openjdk package faster from the dfw mirror via http than rsync via afs00:33
ianwthe cache is working; after i've downloaded it once ...    231,514,016 100%   86.30MB/s    0:00:02 (xfr#1, to-chk=0/1)00:35
*** brinzhang has joined #opendev00:43
ianwfor some reason, nodepool still seems to be trying to upload to openedge00:44
ianwmight have been an old config from when the builders were on hold00:54
ianwi've manually cleaned up the zk nodes00:56
ianwi feel like the builders are in good shape.  they've got free space; images modulo the opensuse-15 issues are all building and there's no dangling images/uploads/etc.00:57
fungicorvus: yay! excellent news01:01
fungipyn38: yes, you're right (as far as i can see)01:05
pyn38fungi. Thanks again! ..It got merged :)01:18
fungigreat! you know where to find us if you have more questions01:24
pyn38Yes. I do! May I know which time zone do you work ?. SO that we know right time to ping.01:26
*** pyn38 has quit IRC02:15
openstackgerritMerged openstack/diskimage-builder master: docs: fix wrong key in the lvm sample configuration
fungidifferent folks in here are from various parts of the world, i personally am on the east coast of north america, eastern standard time02:21
*** hamalq_ has quit IRC02:50
*** pyn38 has joined #opendev04:14
*** larainema has joined #opendev04:19
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: add note about requiring debian-minimal for DIB_APT_SOURCES_CONF
*** ykarel has joined #opendev04:57
*** lpetrut has joined #opendev05:50
*** lpetrut has quit IRC05:54
*** ysandeep|away is now known as ysandeep06:11
openstackgerritPranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin
*** marios has joined #opendev06:20
*** pyn38 has quit IRC06:22
*** lpetrut has joined #opendev06:28
*** lpetrut has quit IRC06:30
openstackgerritPranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin
openstackgerritPranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin
openstackgerritPranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin
openstackgerrityatin proposed zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux
openstackgerritPranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin
openstackgerritDinesh Garg proposed zuul/zuul-jobs master: Allow customization of pip and helm binary
*** eolivare has joined #opendev07:33
openstackgerritDinesh Garg proposed zuul/zuul-jobs master: Allow customization of pip and helm charts
*** slaweq has joined #opendev07:42
*** ralonsoh has joined #opendev08:02
*** sboyron has joined #opendev08:13
*** andrewbonney has joined #opendev08:15
*** ysandeep is now known as ysandeep|lunch08:18
*** rpittau|afk is now known as rpittau08:30
*** sboyron has quit IRC08:38
*** hashar has joined #opendev08:50
openstackgerrityatin proposed zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux
*** elod is now known as elod_pto09:04
*** slaweq has quit IRC09:16
*** slaweq has joined #opendev09:18
*** slaweq has quit IRC09:20
*** ysandeep|lunch is now known as ysandeep09:22
*** slaweq has joined #opendev09:24
openstackgerritPranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin
*** DSpider has joined #opendev10:43
*** slaweq has quit IRC10:51
*** slaweq has joined #opendev10:55
*** ysandeep is now known as ysandeep|afk11:24
*** hashar is now known as hasharLunch11:27
*** brinzhang has quit IRC12:08
*** sboyron has joined #opendev12:10
*** ykarel_ has joined #opendev12:14
*** ykarel has quit IRC12:17
*** ykarel_ is now known as ykarel12:18
openstackgerritTobias Henkel proposed openstack/diskimage-builder master: POC: Pre-pull docker images
*** hasharLunch is now known as hashar12:22
*** sboyron has quit IRC12:33
*** tosky has joined #opendev12:58
*** ysandeep|afk is now known as ysandeep13:00
ykarelreview please
*** mlavalle has joined #opendev13:59
*** ysandeep is now known as ysandeep|away14:22
ykareltobiash, fungi ianw ^14:38
*** iurygregory has joined #opendev14:38
fungiykarel: lgtm, but we should probably discuss in #zuul since zuul-jobs is a zuul project repository15:12
ykarelfungi, ok just joined #zuul15:12
*** rpittau is now known as rpittau|afk15:38
*** hashar has quit IRC15:53
* prometheanfire would like to see gertty get a release, if only because it's been a while and there are a couple of commits that would be nice to have
fungiprometheanfire: that's one of corvus' personal projects, so you might ask him when he's around16:07
corvusprometheanfire: good point, i'll work on that soon16:24
prometheanfirecorvus: yarp, thanks16:27
prometheanfirecorvus: main reason is because I used the git version (because some py38 issue I thought at the time) and now since the DB upgraded I can't go back :P16:28
openstackgerritMerged zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux
*** hashar has joined #opendev16:31
*** gibi is now known as gibi_pto16:35
corvusprometheanfire: i may make a few more gerrit 3.x changes before making a release16:41
openstackgerritMerged ttygroup/gertty master: Add version specific changes for git-url
openstackgerritJames E. Blair proposed ttygroup/gertty master: Document removal of /p from default git-url
openstackgerritMerged ttygroup/gertty master: Document removal of /p from default git-url
openstackgerritMerged ttygroup/gertty master: Fix auth-type in opendev example config
openstackgerritMerged ttygroup/gertty master: tokenizer: do not try to decode strings on Python 3
*** chandankumar is now known as raukadah16:53
*** marios is now known as marios|out17:01
corvuszbr|rover: the big constraints that caused us to decide that slim images are better for opendev: image build/upload time, hitting cloud limits on image sizes, needing newer testing tools, interaction between different tools17:08
zbr|rovermy expectation is that a fat image would not be updated more often than twice a month?17:10
mordredand making sure we were testing as much of service install as possible17:10
mordredmany years ago we had images with things pre-installed and it caused us to miss places where installation automation didn't actually install something17:10
zbr|roveri will try to check how big their images are, is just personal curiosity.17:11
mordredzbr|rover: oh - I see the question - yeah, one of the big differences is tha tgithub acitons (and other similar systems) aren't design to support end ot end integration tests that involve installation of complex software17:12
mordredfat images can work well when the focus is more on unit tests, etc17:12
zbr|rovermordred: YES. obviously that for integration, gha is uselss.17:13
zbr|roverbut for quick tests (lint, unit, functional), it is likely that fat-image approach to be more economical.17:13
zbr|rovermaybe they use some different virtualization that makes them care less about the size of the system images.17:14
corvusthey only have one cloud17:15
mordredyah. and they don't also have the slim images17:15
mordredfor us it would double the number of images we'd need to make- because we have to have the slim ones for integration17:16
corvusso we have local mirrors to make the pre- steps as fast as possible17:16
mordredmany of these things are much easier when you only have one cloud17:16
zbr|roverprobably that for us it would be easier to use containers for simple tests.17:16
mordredespecially when you also run that cloud :)17:16
corvuszbr|rover: sure, we'd just download the container images from our local container image cache.  similar network traffic (probably more), but potentially less install time (depending on the speed of the os package manager)17:17
*** hamalq has joined #opendev17:19
*** hamalq_ has joined #opendev17:21
*** hamalq has quit IRC17:24
*** marios|out has quit IRC17:29
fungito be clear, what we had previously was two sets of images: fat images for things like unit tests and slim images for things like devstack. maintaining twice as many images is also a pain, as is having to debug why things don't install and start breaking your image builds because it's happening outside of testing, or having to reroll images in an emergency because some new release of something you're baking17:33
fungiin no longer works as expected17:33
fungicollapsing all our testing onto a consistent set of images was a lot of work up front, but saved us a ton of maintenance we'd been continuously engaged in trying to support the fat image case17:34
fungiit also allows projects to manage and solve their own dependency problems, rather than relying on us to fix those things when they're included directly in our images17:35
fungialso our fat images were an echo of an even earlier model, where we reused persistent nodes for lightweight jobs like unit testing, so the "fat" images were to some degree a transitional step from those persistent nodes which had lots of tools preinstalled on them to images with mostly the same sets of tools to avoid breaking the jobs as we moved to more and more ephemeral nodes17:37
mordredfungi: ah - the good old days where the reused persistent nodes had all the prerequsites maintained via 'apt-get build-dep $service' - so the process for updating deps was to first land a change to the _packaging_ which would then cause the node maint to update the packages on the test node17:43
mordredthat system, horrible as it was, mostly worked without intervention on my part for almost a year- until it completely fell apart17:43
mordred(markmc especially did not like that system since it required him to update debian packaging in order to manipulate depends :) )17:45
openstackgerritJames E. Blair proposed ttygroup/gertty master: Handle gerrit 3.x style internal urls
corvusthat makes clicking links work again for me17:46
corvus(i had been typing in numbers)17:46
fungicorvus: thanks! i was seeing similar problems and thought maybe i'd done something wrong17:49
*** ralonsoh has quit IRC17:57
*** eolivare has quit IRC18:14
*** ykarel has quit IRC18:28
*** hashar is now known as hasharDinner18:31
*** andrewbonney has quit IRC18:33
openstackgerritMerged zuul/zuul-jobs master: Create tox_package_name for tox role
openstackgerritJames E. Blair proposed ttygroup/gertty master: Add WIP support
*** hasharDinner is now known as hashar19:02
openstackgerritJames E. Blair proposed ttygroup/gertty master: Add WIP support
corvusfungi, prometheanfire: ^ that should come in handy too19:02
fungioh, yup!19:03
*** dviroel has quit IRC19:08
fungithat ternary assignment trick never gets old, which i could remember to use it19:13
fungiit's so c-like19:14
corvusthat's why i like it :)19:15
*** mlavalle has quit IRC19:48
hasharhi corvus, I have send a config to setup the plugins/zuul-results-summary on Google Gerrit . Does that need a googler to approve / deploy?20:03
openstackgerritmelanie witt proposed opendev/elastic-recheck master: Add query for bug 1882521
openstackbug 1882521 in OpenStack Compute (nova) ussuri "Failing device detachments on Focal" [Undecided,New]
melwitt^ an effort to improve our categorization rate20:28
melwitt*classification rate20:29
*** slaweq has quit IRC20:54
*** Alex_Gaynor has joined #opendev21:16
Alex_GaynorIs anything going on with the arm64 builders for OpenDev? I'm seeing jobs that haven't started for a while:
fungii can take a look once i'm done cooking dinner. hopefully that linaro cloud hasn't fallen offline again21:31
fungiAlex_Gaynor: not offline i don't think, graphs show we're using nodes there, but only using roughly half the max-servers we set:
Alex_Gaynorfungi: this is just eyeballing the graphs, but it looks like a non-trivial percentae of node launches failed?21:46
fungiyeah, i concur, i haven't pulled up the launcher logs yet but the low utilization suggests nodepool is probably getting its boot requests rejected for being at/over quota (could happen if there are node leaks in the provider, for example)21:48
fungiin the debug log i'm seeing errors booting in linaro-us due to timeouts (nodes not becoming ready after waiting)21:53
fungikevinz: ^ not sure if you're around, but is something overloaded there?21:54
fungiutilization has fallen even more but i still see changes waiting in the check-arm64 queue at too. back to the logs now that dinner is done22:37
*** hashar has quit IRC22:45
fungithe launcher is logging occasional timeouts like this but it's not as if there's a constant stream of them:
fungihuh, well it apparently caught up after all, i'm not sure why it managed to clear the queue of pending kolla and kolla-ansible changes which were missing node assignents22:53
fungier, assignments22:53
fungii don't see anything queued for either22:54
*** mlavalle has joined #opendev22:56
fungithree occurrences of boot timeouts in the past two hours23:07
ianwodd; unfortunately not much we can do but report upstream23:11
fungiyeah, i wasn't able to pinpoint why node requests were either not being seen by the launcher or not being claimed23:14
fungibut it eventually worked through whatever was blocking it23:14
*** dmsimard5 has joined #opendev23:50
*** dmsimard has quit IRC23:52
*** dmsimard5 is now known as dmsimard23:52
*** DSpider has quit IRC23:57

Generated by 2.17.2 by Marius Gedminas - find it at!