Wednesday, 2023-01-18

opendevreviewMerged openstack/project-config master: linaro-us : set max-servers to 0
clarkbit does look like networking to that mirror node is unhappy for some reason00:01
ianwnothing interesting on the console log00:07
ianwPING ( 56(84) bytes of data.00:09
ianwFrom icmp_seq=2 Redirect Host(New nexthop:
ianwinteresting response00:09
fungithat does look like a routing issue00:18
fungiis that the new environment or the old one?00:18
ianwthat's the new one00:23
ianwone tangential thing i've noticed is that css is broken on older archives -> e.g.
ianwthe css links are http, which my firefox refuses to load from the https site00:23
ianwanyway the instanance shows a power state of "paused" ... but i don't think it is00:24
ianw OS-EXT-STS:power_state      | Paused  00:30
ianw OS-EXT-STS:vm_state         | active00:30
ianwit's both active and paused00:30
ianwkevinz: ^ if around00:31
ianwfungi: do you see that?  I only seem to see that icmp redirect if i ping from RAX00:36
ianwI have rebooted the server, and it is now active.  i don't know what happened to it :/00:37
*** rlandy is now known as rlandy|out00:45
ianwthere seems to be plenty of arm64 jobs queued, but i can't see that nodepool is choosing to try and start nodes on the new cloud ....00:51
Clark[m]Does grafana show errors?01:01
opendevreviewIan Wienand proposed opendev/system-config master: make-tarball: role to archive directories
opendevreviewIan Wienand proposed opendev/system-config master: tools/
opendevreviewIan Wienand proposed opendev/system-config master: bridge: Install common key and test tarball artifact generation
ianwClark[m]: it doesn't seem to get that far even01:09
ianw2023-01-18 01:05:06,011 INFO nodepool.PoolWorker.linaro-regionone-main: [e: 2686ac2061784e33b808c4385fa5e9f3] [node_request: 300-0020204239] Declining node request 01:09
ianw'node_types': ['ubuntu-jammy-arm64', 'ubuntu-bionic-arm64', 'ubuntu-focal-arm64', 'ubuntu-jammy-arm64']01:10
ianwthis was a request for some nodes it should have been able to provide01:10
ianwohhhh, i bet this is a quota issue01:16
*** dasm is now known as dasm|off01:24
fungiianw: whether you see nexthop responses is going to depend on what icmp types your firewall allows to arrive01:36
fungior whether they're being filtered upstream of you01:37
opendevreviewClark Boylan proposed opendev/system-config master: Switch Gerrit to Java 17
*** yadnesh|away is now known as yadnesh04:43
opendevreviewDaniel Blixt proposed zuul/zuul-jobs master: Reset connection before testing build ssh-keys
opendevreviewDaniel Blixt proposed zuul/zuul-jobs master: Allow mirror push to delete current branch
opendevreviewDaniel Blixt proposed zuul/zuul-jobs master: Make build-sshkey handling windows compatible
*** jpena|off is now known as jpena08:21
*** yadnesh is now known as yadnesh|afk08:26
*** ysandeep is now known as ysandeep|lunch08:44
*** soniya29|rover is now known as soniya29|rover|lunch08:55
*** soniya29|rover|lunch is now known as soniya29|rover10:06
*** ysandeep|lunch is now known as ysandeep10:36
*** rlandy|out is now known as rlandy11:11
*** dviroel|afk is now known as dviroel11:19
*** dviroel|afk is now known as dviroel11:54
*** ysandeep is now known as ysandeep|brb12:33
*** ysandeep|brb is now known as ysandeep12:45
dpawlikfungi, clarkb o/ 13:01
dpawlikseems that Facebook fedora mirror is not working as expected13:02
dpawlik is empty13:02
dpawlikthey have move it to ...13:03
*** gthiemon1e is now known as gthiemonge13:09
fungidpawlik: i think that's a fedora change, not facebook?13:15
fungifedora periodically "archives" their old releases13:15
fungiprobably means we're behind in moving our fedora images to a newer version13:15
Tenguah, yeah, 35 is eol13:16
dpawlikyeah, that's it13:16
dpawlikdid not read the README 13:17
dpawlikthanks fungi13:17
*** soniya29|rover is now known as soniya29|rover|brb13:20
*** dasm|off is now known as dasm13:34
*** ysandeep is now known as ysandeep|afk14:54
*** dviroel is now known as dviroel|lunch15:32
*** dviroel|lunch is now known as dviroel16:23
clarkbfungi: frickler: is an easy one to drop gerrit 3.5 images16:41
clarkbfrickler: and then should fix our 3.6 images (they can land in either order though removing 3.5 first runs fewer jobs in total)16:42
fungiyeah, i'm stepping through that series now. thanks!16:43
clarkbwith those two changes in I might end up merging my add 3.7 images stack and java 17 stack. I think we should probably move to java 17 first?16:43
clarkb*with those two changes landed16:43
clarkbactually I guess the order there doesn't matter a whole lot. We can add 3.7 then java 17 and it will just apply to both 3.6 and 3.716:44
fungii approved the 3.7 images change too. should i wait on the upgrade job or will it complicate the java 17 work?16:45
amorinhello opendev team, we are struggling with a test on mistral, about dnspython, we are not sure how to fix it, would you mind giving us a hint to follow good rules?16:46
amorinthis is what we proposed so far, but we have the feeling it's wrong16:46
clarkbfungi: I don't think it will. since both 3.6 and 3.7 can do java 1716:46
fungiamorin: i have a feeling it will be very similar to the one swift had to fix yesterday. will look as soon as the tc meeting wraps up16:46
amorinack, thanks!16:46
clarkbfungi: I also had to implement a hack to make java 17 work due to its interesting thatthey say they fully support java 17 in 3.6 then this happens16:47
fungiamorin: in fact, it's identical. right down to only showing up in the docs builds16:47
fungithe problem is that testenv:docs installs the documentation build requirements with constraints, but tox is also installing mistral's dependencies in a separate step and no constraints are app16:48
amorinyes, a fix in tox.ini maybe?16:49
fungiyes, exactly, the solution the swift team went with was to add an explicit requirements.txt as well in the deps list for testenv:docs16:49
amorinok, will check swift gerrit logs to figure that out16:50
fungii'll find you a link to their change16:50
fungiyes that's the one16:50
opendevreviewCorey Bryant proposed zuul/zuul-jobs master: Adapt tox_extra_args to tox 4.x
opendevreviewMerged opendev/system-config master: Remove Gerrit 3.5 images
*** marios is now known as marios|out17:08
opendevreviewMerged opendev/system-config master: Convert Verified MaxWithBlock to submit-requirement in testing
*** yadnesh|afk is now known as yadnesh|away17:13
frickleramorin: fyi seems eventlet has released a fixed version and I've proposed an u-c bump now . fixing your doc builds still is a good thing17:25
*** gmann is now known as gmann_afk17:29
*** rlandy is now known as rlandy|laptop_return17:38
*** gmann_afk is now known as gmann17:41
fungiyeah, there's basically two issues there. one is the dnspython+eventlet incompatibility, but the other is that the docs builds for some projects weren't correctly applying constraints17:41
*** jpena is now known as jpena|off17:42
opendevreviewMerged opendev/system-config master: Add Gerrit 3.7 images
*** gmann is now known as gmann_afk18:06
*** rlandy|laptop_return is now known as rlandy18:32
*** gmann_afk is now known as gmann18:56
amorinack frickler, thanks for info, we applied the patch like swift team and that worked19:38
*** prometheanfire is now known as Guest176920:12
*** dviroel is now known as dviroel|afk21:09
*** Guest1769 is now known as prometheanfire21:24
ianwok, nl03 is now reporting
ianwbah, it is not, that is my copy buffer22:04
ianwit is reporting "Compute service reports fault: No valid host was found."22:05
clarkbthis is trying to boot in the new cloud?22:06
clarkbcould be a placement issue if things are not accounted for properly basically it thinks there is no room when there is ?22:06
ianwyeah it seems hard to diagnose from this side.  or we're somehow asking for something it doesn't have22:07
ianwi can start a m1.medium with a standard image22:16
ianwand we do have opendev-control node running ... the common theme i'm sensing here here is that both don't have ephemeral disk22:17
ianwwhile the ci node flavor does22:17
ianwok, if i try to boot an image with 8cpu/8gb/no ephemeral i get a different error 22:39
ianwExceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 22:39
ianw... but if i do the same with the plain ubuntu-22.04 image i *don't* get a failure22:41
ianw... so something about the nodepool uploaded images22:42
ianwubuntu-22.04 is raw, and we're uploading .qcow2.  i wonder if we want raw22:54
ianwi think what i need to do here is shutdown nb03 and let nb04 upload raw images.  we already upload raw to osuosl23:08
ianwshutting down nb03 is just so that i don't have to worry about anything parallel uploading23:09
*** rlandy is now known as rlandy|out23:19
ianw... interesting, nb03 has shutoff itself anyway, along with the mirror 23:21
fungihow... fun23:21
ianwi asked kevinz to cleanup some leaked hosts there, which he reported he did, i imagine that had something to do with it23:22
ianwi've put both nb03 and the linaro-us mirror in emergency.  the region is already turned off with max-nodes: 0 from yesterday23:22
opendevreviewIan Wienand proposed openstack/project-config master: nb04: use linaro region mirror
opendevreviewClark Boylan proposed opendev/system-config master: Update git in gitea images

Generated by 2.17.3 by Marius Gedminas - find it at!