Sunday, 2021-01-17

*** tosky has quit IRC00:22
*** DSpider has quit IRC00:26
*** brinzhang_ has quit IRC02:39
fungimeh, i'm really not going to catch it any more idle, going ahead with the restart momentarily02:57
fungi#status log restarted zuul scheduler, now gerrit wip state should prevent zuul from attempting to merge03:03
openstackstatusfungi: finished logging03:03
fungii'll reenqueue the previously running changes as soon as it's done collecting configuration03:04
fungilooks like it's done, but zuul-web seems unhappy now... i just get proxy errors from apache03:08
fungii'll give it a minute before i restart it too03:09
fungino response on
fungirestarting it03:11
funginow it works again03:12
fungi#status log restarted zuul-web because 9000/tcp became unresponsive after the scheduler restart and did not recover on its own03:12
openstackstatusfungi: finished logging03:12
fungireenqueuing now03:13
fungiand done (there were only 6 changes in total anyway)03:13
fungii'll try to keep an eye on things for a little while before i disappear for the night03:13
*** paladox has quit IRC03:30
*** paladox has joined #opendev03:30
*** jrosser has quit IRC04:29
*** jrosser has joined #opendev04:29
openstackgerritDr. Jens Harbott proposed zuul/zuul-jobs master: Temporarily stop running Gentoo base role tests
*** brinzhang has joined #opendev07:32
*** paladox has quit IRC08:04
*** dmellado has quit IRC08:04
*** paladox has joined #opendev08:04
*** dmellado has joined #opendev08:04
*** slaweq has joined #opendev08:20
prometheanfirefungi: I did find a small issue it looks like, a couple08:31
prometheanfire2021-01-17 08:17:44.333 | Image file test-nodepool-image-20210117.qcow2 created...08:31
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: add missing packages for python reinstall on gentoo
openstackgerritMatthew Thode proposed openstack/project-config master: allow create-repo-list to be run without six
openstackgerritMatthew Thode proposed openstack/project-config master: add gnupg2 to package map for gentoo
prometheanfirethose three reviews should fix it, I think08:41
prometheanfirefixed it for me at least08:41
*** brinzhang has quit IRC09:16
*** slaweq has quit IRC09:57
*** DSpider has joined #opendev09:59
*** slaweq has joined #opendev10:27
*** slaweq has quit IRC10:29
*** zoharm has joined #opendev10:36
zoharmHi all, I would like to ask for some pointers regarding Cinder driver 3rd party CI.10:48
zoharmWe have devstack streamlined to use our storage backend and are able to launch successful tempest runs. My question/request is, what are useful resources documenting the integration points needed for Gerrit, initiating assigned CI runs, and publishing results?10:48
zoharmFor example, is a hand crafted CI server a viable option? That would listen to the Gerrit event stream, pick up patchset-create events, launch devstack with the commit, run tempest, upload artifacts, and leave a gerrit review comment.10:48
zoharmAny more recommendatinos or any feedback please?10:48
*** slaweq has joined #opendev11:04
*** tosky has joined #opendev11:50
*** zoharm has quit IRC12:34
*** zoharm1 has joined #opendev12:34
*** zoharm1 has quit IRC13:52
*** zoharm has joined #opendev13:52
*** avass has quit IRC14:08
*** yourname has joined #opendev14:09
*** yourname has quit IRC14:14
*** yourname has joined #opendev14:18
fungizoharm: yes, you could create a ci system from scratch. note that we don't dictate what you should test or report on, that's up to the cinder folks. but be aware that there are a lot of nuances when it comes to testing proposed changes, especially when a change can depend on another change, perhaps in a completely different project, and testing them together on current branch states rather than the states14:29
fungithe branches were in for the developer's local checkout... we solved a lot of this by creating
fungibut if you want to make a ci system yourself from scratch, we mainly just ask that you make sure it doesn't generate an unnecessary volume of queries against gerrit, that it only reports on changes for projects which have requested that of you, and you'll probably want to make sure it reports results in the same format as zuul does within review comments because we have integration to be able to parse and14:30
fungisummarize those for reviewers14:31
*** iurygregory has joined #opendev14:31
fungiprometheanfire: i've approved those, though the dib change is going to need a release before we can use it, i think?14:37
openstackgerritMerged openstack/project-config master: allow create-repo-list to be run without six
openstackgerritMerged openstack/project-config master: add gnupg2 to package map for gentoo
*** SWAT has joined #opendev15:11
openstackgerritMerged openstack/diskimage-builder master: add missing packages for python reinstall on gentoo
*** zoharm has quit IRC15:32
*** zoharm1 has joined #opendev15:32
*** tosky_ has joined #opendev16:05
*** tosky has quit IRC16:05
*** tosky_ is now known as tosky16:06
*** sshnaidm|afk has quit IRC16:12
*** sshnaidm|afk has joined #opendev16:27
*** sshnaidm|afk has quit IRC16:32
fungiianw: frickler: ^ when you're around next, opinions on a quick dib release for that?16:33
*** sshnaidm has joined #opendev16:54
*** stevebaker has joined #opendev19:00
*** slaweq has quit IRC19:04
*** slaweq has joined #opendev20:55
*** slaweq has quit IRC20:56
ianwi'll check out the queue to see what state we're in21:16
ianwok, mirror-update has a bunch of stuck publish-mirror-logs21:25
ianwthat's an odd file21:25
fungiindeed, could it be a side effect of me holding locks for all the mirrors21:26
ianwnot sure, something must have gone crazy with rotation21:26
fungibtw, there's another project.tarballs vos release in progress since 01:55 utc, doesn't claim to be a full release this time, probably it's whatever changed recently, but i expect it's slowed by the other transactions still lingering21:27
fungivos status on afs01.dfw claims there are presently 8 transactions in progress21:28
fungithat almost looks like there were multiple logrotate invocations in quick succession or something21:30
fungior at the same time?21:30
openstackgerritIan Wienand proposed opendev/system-config master: reprepro: only rotate .log files
ianwfungi: ^ i think that might be it21:31
ianwi've killed all those and will remove anything that's not a .log file21:33
fungioh, nice. thanks21:34
fungii wonder why it's only happening recently... i guess we redid the logrotate there not too long ago?21:35
ianwnot sure yet ...21:37
ianwrunning it manually21:37
ianwrsync: mkstemp "/afs/" failed: File too large (27)21:37
ianwohh, it might make sense why things were hanging if that volume is full21:39
ianwmirror.logs                100000000      58824    0%         82%21:41
ianwi wonder though if it's file name/path length though21:41
* fungi will assume docker folks just don't understand the relationship between debian and ubuntu21:42
fungialso we can probably try to end any in-progress transactions if they're not for project.tarballs21:43
ianwi feel like this might be related to the large number of very long filenamed files21:43
fungioh, yeah, there's a file record limit which varies depending on the length of the filenames21:43
ianwi'm just going to rm from the afs partition too21:43
ianwok a manual run of /usr/local/bin/publish-mirror-logs works now21:46
ianwso i see 8 busy volumes21:47
fungione of which is the project.tarballs volume with a release underway21:49
fungithe other 7 are associated with vos releases for mirror volumes which were manually killed, but the transactions weren't ended because at the time we had no available slots to execute rpc calls21:51
ianwinterestingly more than that are showing as locked21:52
ianwall the following are locked
ianwseveral of those are locked, but don't show up in the busy list of volume21:54
fungiyeah, likely we'll need to delete some locks on volumes with no running transactions21:56
ianwi'll just try clearing all these afs locks and see if we can get things back in sync22:01
ianwi will leave tarballs alone22:01
fungithanks, tarballs should be locked, there's an actual vos release running (it's just fighting for bandwidth with those other vestigial transactions which haven't ended on their own yet)22:03
ianwthey're unlocked now, but the volumes are still busy22:07
*** zoharm has joined #opendev22:08
*** zoharm1 has quit IRC22:11
ianwfungi: are you sure the tarballs release is currently doing anything?  rxdebug seems to show afs02.dfw idle22:12
fungiit's possible the tarballs release is waiting on other transactions to end22:13
ianwvos status shows the active transactions22:17
fungiyeah, mapping those to the volume ids mentioned by vos listvldb, 536871090 is the read-only replica for project.tarballs22:18
fungithat's the first transaction returned for me by vos status22:19
ianwi feel like we should endtrans the rest of them ...22:19
ianwit seems to me the volume list was not in sync with the transactions anyway22:20
ianwlike volumes were locked that did not have busy volumes (and presumably transactions running)22:20
fungithe others are read-only volumes for mirror.ubuntu, mirror.fedora, mirror.centos, mirror.ubuntu-ports, mirror.opensuse, mirror.debian, and mirror.yum-puppetlabs22:21
fungithose kicked off via cron before i thought to hold the locks for them22:21
ianwthat correlates with
fungialso i don't think afs02.dfw is idle, based on the eth0 traffic graph for it in cacti at least22:22
ianwno i agree, rxdebug must just be saying it can respond22:22
*** DSpider has quit IRC22:42
fungii've submitted a vos endtrans on those other 7 transactions22:55
ianwi think it can take 10+ minutes for anything to happen, if it does.  something about slots timing out22:55
fungithough i don't know whether it will abort them22:55
ianwi'm looking at the ord server with an eye to trying our 1.8 manual update22:56
ianwand moving bits into ansible as i can22:56
fungimight as well, sure22:56
ianwthe only thing we have on there is docs22:57
ianwit has a bunch of other readonly volumes that the volume database doesn't think it has22:58
ianwi.e. vos listvol -server shows them, but vos listvldb doesn't show them22:59
ianwi'm going to zap the volumes of afs01.ord that don't have vldb entries to clean this up before attempting an upgrade23:11
fungisounds fine to me, than23:12
fungifwiw, none of the transactions i endtrans'd have budged, still showing up in vos status23:13
ianwmirror.deb-hammer.readonly        536870959 RO    1940675 K On-line23:14
ianwmirror.npm.readonly               536870941 RO  609696982 K On-line23:14
ianwmirror.pypi.readonly              536870932 RO  298510820 K On-line23:14
ianwmirror.ubuntu.readonly            536870950 RO  156858708 K On-line23:14
ianwmirror.wheel.cent7x64.readonly    536870956 RO     443230 K On-line23:14
ianwmirror.wheel.trustyx64.readonly   536870947 RO     450824 K On-line23:14
ianwfor reference it is ^ on ORD23:14
*** tkajinam has quit IRC23:17
*** tkajinam has joined #opendev23:17
ianwi'm half feeling we should pull 1.8.7 openafs into our ppa23:20
ianwjust to start there.  but not sure23:20
*** tosky has quit IRC23:21
*** tosky_ has joined #opendev23:21
*** tosky_ is now known as tosky23:21
fungiseems like a reasonable choice. i guess 1.9 is still a bit green23:22
ianwsalsa doesn't have it, so might as well stick with what's there23:27
fungiyeah, salsa's only going to have what the debian package maintainer for openafs has pulled to work on packaging23:30
fungiwouldn't be surprised if 1.9 isn't even on their radar yet from that perspective23:30
ianwyep, 1.8.6-5 was pushed as the emergency update into debian23:32
ianwso may as well stick there23:32
ianwremoving these old npm and pypi volumes on ord is taking it's time23:33
fungithe other 7 transactions at least have "transactionFlags: delete" now according to vos status23:36
openstackgerritMerged opendev/system-config master: reprepro: only rotate .log files

Generated by 2.17.2 by Marius Gedminas - find it at!