Friday, 2021-01-08

*** paladox has quit IRC00:13
*** paladox has joined #opendev00:15
*** JayF has quit IRC00:33
*** mlavalle has quit IRC02:06
*** _mlavalle_1 has joined #opendev02:07
*** ysandeep|away is now known as ysandeep03:19
*** hamalq has quit IRC04:01
*** ykarel has joined #opendev05:05
*** ysandeep is now known as ysandeep|afk05:08
*** whoami-rajat__ has joined #opendev05:25
*** marios has joined #opendev06:46
*** ralonsoh has joined #opendev07:03
*** dirk has quit IRC07:09
*** dirk has joined #opendev07:10
*** fressi has joined #opendev07:27
*** noonedeadpunk has joined #opendev07:39
*** ysandeep|afk is now known as ysandeep07:42
*** fressi has quit IRC07:46
*** eolivare has joined #opendev07:46
*** fressi has joined #opendev07:58
*** hashar has joined #opendev08:07
*** fressi has quit IRC08:21
*** andrewbonney has joined #opendev08:24
*** rpittau|afk is now known as rpittau08:32
*** fressi has joined #opendev08:35
*** ykarel is now known as ykarel|lunch08:39
*** tkajinam has quit IRC08:55
*** tosky has joined #opendev08:56
*** fressi has quit IRC09:12
*** fressi has joined #opendev09:17
*** ysandeep is now known as ysandeep|lunch09:22
*** hashar has quit IRC09:28
*** hashar has joined #opendev09:28
*** brinzhang has quit IRC09:38
*** ykarel|lunch is now known as ykarel09:54
openstackgerritdaniel.pawlik proposed openstack/diskimage-builder master: Replace Fedora 31 functional tests with Fedora 33
*** DSpider has joined #opendev09:57
*** dtantsur|afk is now known as dtantsur09:57
*** whoami-rajat__ has quit IRC10:13
*** ttx has quit IRC10:16
*** ttx has joined #opendev10:17
*** hashar is now known as hasharLunch10:43
*** ysandeep|lunch is now known as ysandeep11:32
*** hasharLunch is now known as hashar12:32
*** SotK has quit IRC12:52
*** SotK has joined #opendev12:52
*** whoami-rajat___ has joined #opendev13:18
*** stzsch has quit IRC13:38
*** _mlavalle_1 has quit IRC13:58
*** mlavalle has joined #opendev13:58
*** dtantsur is now known as dtantsur|brb14:00
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation.
danpawlikianw_pto: hey. I'm trying to reproduce error for dib-nodepool-functional-openstack-fedora-33-src  CI job in and normally all is working correctly. Maybe you will see something strange14:11
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation.
*** rpittau is now known as rpittau|afk14:46
*** ysandeep is now known as ysandeep|away14:49
fungidanpawlik: i can hopefully take a look after i finish checking over our afs volume replicas for any more fallout from yesterday14:53
*** fressi has quit IRC14:54
danpawlikack fungi, thank you14:54
*** fressi has joined #opendev14:54
*** fressi has left #opendev14:55
fungitarballs site volume has been releasing for the past 8 minutes, should be better shortly i think15:08
fungi#status log killed a hung vos examine process from yesterday's afs issues which was preventing static site volume releases from proceeding15:21
openstackstatusfungi: finished logging15:21
danpawlikfungi: indeed, but why it happens so. Normally in f33 is rpm is in 4.1615:37
danpawlikfungi: but on buildroot which is bionic is 4.14.115:38
danpawlikfungi: but I'm not so familiar with DIB project15:39
fungidanpawlik: seems like maybe we need a newer rpm package for ubuntu... even focal is still carrying rpm 4.14.1:
fungipresumably because that's the newest in debian as well:
openstackDebian bug 979062 in src:rpm "please update to rpm 4.16.x" [Important,Open]15:43
*** dtantsur|brb is now known as dtantsur15:45
danpawlikthanks fungi++15:46
danpawlikfungi: I was thinking that Im doing something wrong15:46
danpawlikfungi: could you comment in the PS or should I do :>?15:47
fungidanpawlik: this may be the impetus to go ahead with mordred's previous suggestions of switching to bootstrapping images from existing docker images instead of bootstrapping into a completely empty chroot15:51
fungipresumably there's a f33 x86_64 docker image which contains the correct version of rpm already15:52
clarkbfungi: re the vos examine that was on the side where we do vos release? and not a lock on the server side we needed to break?15:57
fungiclarkb: correct15:57
clarkbfungi: danpawlik aiui ianw is still working on the chroot bootstrap from docker images15:57
clarkbnot sure what is needed next there though15:57
fungithe cronjob on mirror-update.o.o to run the static release script every 5 minutes and it does a vos examine before releasing each volume, that was indefinitely waiting for a response from afs01.dfw from back when the server was hung yesterday. the replicas weren't locked and seemed intact15:59
fungionce i killed the vos examine, the next run of the cronjob seems to have worked normally (albeit took a while because of the backlog of writes)15:59
clarkbgot it16:00
mordredfungi: I think using docker images as the source for the base chroot for dib would get us a long way16:09
mordredit's a non-zero amount of work still because of some of dib's assumptions about how root elements work and what they provide - but it's also still doable16:09
clarkbyup I think simple testing of it is owrking now16:09
*** jpena has joined #opendev16:15
*** hashar has quit IRC16:25
*** ykarel_ has joined #opendev16:26
*** ykarel has quit IRC16:29
*** whoami-rajat___ is now known as whoami-rajat__16:39
*** marios is now known as marios|out16:44
*** marios|out has quit IRC16:53
prometheanfirefungi: clarkb can we cut a release of glean?16:54
fungiprometheanfire: seems timely, sure16:55
clarkbya I'm not opposed other than it is friday. But we can trigger a nodepool image build early today once that publishes if we need to16:56
clarkbfungi: looks like we have 2 bug fixes so would be 1.18.2 ? Do you want to push that tag or should I go get my keys16:57
fungii can in a few minutes, sure16:58
fungi1.18.2 seems reasonable given the merged commits since 1.18.116:58
*** jpena is now known as jpena|off17:02
*** ykarel_ is now known as ykarel17:21
fungiglean 1.18.2 is up on pypi17:39
*** sboyron has quit IRC17:39
fungi#status log released glean 1.18.217:39
openstackstatusfungi: finished logging17:40
fungiprometheanfire: clarkb: ^17:40
*** sboyron has joined #opendev17:41
ttxfungi, clarkb: do you know what the current state of our ARM support is?17:42
ttxDo we have ARM hosts in nodepool at the moment?17:42
*** ykarel has quit IRC17:42
clarkbttx: we do, they are arm64 vms running on top of a cloud supplied by linaro17:44
ttxok cool. thanks!17:44
clarkbttx: opendev runs some testing on it for the stuff we deploy into the arm64 cloud. Kolla is also doing testing on it iirc. And we do python crptography wheel builds for arm6417:44
*** JayF has joined #opendev17:45
clarkbthere isn't huge capcaity and it is provided by a single cloud so all the caveats of outages and such apply, but it does work17:45
ttxclarkb: was there attempts to get capacity from vexxhost ARM cloud, for redundancy? Or is it completely different17:46
clarkbI don't think we have tried. We've largely relied on folks that grok arm and its requirements to help drive it (linaro really)17:47
ttxok, makes sense17:47
clarkbfungi: prometheanfire: I see new glean on pypi17:55
clarkbI was trying to infer if any images are due for a rebuild soon off of but even sorting by last modified doesn't make that very clear.17:58
clarkbaha a debian buster run is happening now17:59
clarkbif you go to / you see the build logs and the sort there does help17:59
clarkb that image should use the new glean17:59
*** andrewbonney has quit IRC18:01
fungittx: notable about the linaro cloud is that the entire deployment of openstack there is running on arm hardware, not just compute hypervisor hosts18:11
*** eolivare has quit IRC18:14
*** ralonsoh has quit IRC18:17
clarkbthat buster build looks successful but isn't complete yet (it is hashing the resulting artifacts)18:32
clarkbonce hte image is built then it needs to be uploaded. Then we can manually boot one or see if nodepool has successfully booted buster18:32
*** dtantsur is now known as dtantsur|afk18:43
clarkband the build completed successfully a few minuts ago so ya in a few hours I guess we double check glean still works on buster18:43
clarkb021-01-08 18:18:00.277 | Successfully installed glean-1.18.2 <- ocnfirms it used the new glean too18:43
*** whoami-rajat__ has quit IRC18:58
*** hamalq has joined #opendev19:10
*** hamalq has quit IRC19:33
*** hamalq has joined #opendev19:33
*** stzsch has joined #opendev20:31
*** sboyron has quit IRC20:36
openstackgerritlotorev vitaly proposed zuul/zuul-jobs master: Clarity tox_environment accepts dictionary not list
openstackgerritlotorev vitaly proposed zuul/zuul-jobs master: Document Python siblings handling for tox role
*** diablo_rojo__ has joined #opendev20:59
*** slaweq has quit IRC21:13
prometheanfireclarkb: fungi thanks :D21:13
clarkbprometheanfire: if you can confirm it is working as expected for you too that would be great. I'm about to actually ssh into nodes we're running that shoul be updated21:14
prometheanfireya, I'm going to build images21:15
*** cloudnull has quit IRC21:18
clarkbdebian-buster 0022454355 was booted in limestone with the new glean and seems to work21:19
clarkbI checked that by running /usr/glean/bin/pip freeze to get the glean version and being able to ssh into it implies that basic functionality is working21:19
clarkbone thing I'm not able to easily check is the /etc/hosts contents bceause it is running an openstack ansible job whihc changes the hostname and stuff in /etc/hosts21:19
*** beep1560 has joined #opendev21:19
clarkband nodepool just cleaned it up21:20
clarkbI don't think any other images have updated since glean release happened. But buster working is probably a good enough canary21:21
*** hashar has joined #opendev21:24
*** beep1560 has quit IRC21:25
*** cloudnull has joined #opendev21:28
*** hashar has quit IRC21:29

Generated by 2.17.2 by Marius Gedminas - find it at!