Friday, 2021-09-03

*** ysandeep|out is now known as ysandeep06:04
*** ykarel_ is now known as ykarel06:56
*** bhagyashris__ is now known as bhagyashris07:27
*** jpena|off is now known as jpena07:37
*** ykarel is now known as ykarel|lunch08:36
*** ykarel|lunch is now known as ykarel10:07
opendevreviewMerged openstack/project-config master: Set empty nodepool resource lists on inap
*** dviroel|out is now known as dviroel11:21
*** jpena is now known as jpena|lunch11:38
*** jpena|lunch is now known as jpena12:39
dtantsurhi folks! is there a way to opt-out of PDF builds in openstack-docs?13:36
fungithere's probably a switch, but also worth bringing up with the tc i think since there was talk of "pdf docs" being a yoga or z cycle goal13:49
fungiwhich job are you running which is trying to build pdfs? i'll look at it and see what's involved in making it skip that part13:50
fungii know there's a bunch of openstack projects not building pdfs (yet anyway) and their docs jobs are still working13:51
dtantsurfungi: we have a lot of garbage in the rendered pdf, and an addition in my patch seems to cross some threshold after which the PDF refuses to render13:51
dtantsurthe error output is absolute garbage, but I have a guess that I can remove a few images13:52
dtantsurwhich is fine for now, but if the renderer starts to force us to remove important content...13:52
dtantsurfungi: check out page 605 and on:
fungiwell, i was just asking for the job name so i can find where it decides whether to make pdfs, but i can extract it from that raw logset13:53
dtantsuryeah, sorry, I was finishing my complaint13:53
dtantsurit's openstack-tox-docs13:53
fungiaha, it's openstack-tox-docs right there in the url, yep13:54
fungidon't feel you need to complain to me, i didn't make that job, and i honestly don't care if openstack has pdf documentation or not (though it seems others do) ;)13:54
fungibut yeah, i'll see if i can spot how to turn it off13:54
dtantsuryeah, I don't always know where to come with such things :)13:55
dtantsurre TC goal: is there already a proposal to -1? :)13:55
funginot sure, i wasn't following all that closely13:55
fungidtantsur: judging from you could probably just remove the [testenv:pdf-docs ] section from the tox.ini file, but i'll dig deeper and see if there's a rolevar or something you can set in a job variant instead13:58
fungidtantsur: bingo...
dtantsurokay, thanks! we'll use it as the last resort if nothing helps14:00
fungibut yeah, if you add a openstack-tox-docs variant in your check/gate pipelines and set tox_skip_pdf=true in a vars list for it, that should stop trying to build pdfs14:02
fungidtantsur: here's an example where it's being done in a job inheriting from openstack-tox-docs, but you should be able to do similarly in your project-pipeline with an openstack-tox-docs variant since it's not marked final:
fungithough i'll note that codesearch doesn't turn up any projects doing it in their master branches, at least, so there might be a reason14:04
Clark[m] looks like it was a train goal14:06
fungiwhich was never completed i suppose14:11
*** ykarel is now known as ykarel|away14:44
yoctozeptoinfra-root I have a build that seems to have hung
yoctozeptonow it timed out14:52
fungiyoctozepto: i guess it was on post-run tasks? the job looks like it had already failed with the script exiting 114:56
yoctozeptofungi: yeah, that failure was expected; it hang on collecting the logs etc.14:57
fungiinteresting, could you tell what it was running exactly or what the kast task name was?14:58
fungilast task name14:58
yoctozepto2021-09-03 14:22:26.982141 | LOOP [fetch-output : Collect logs, artifacts and docs]14:58
yoctozeptoprimary and secondary2 gave ok14:58
yoctozeptoso it seems secondary1 was lagging14:58
fungii wonder if ansible was unable to reach it14:59
yoctozeptofor half an hour14:59
fungii think it will wait for a very long time for ssh to time out14:59
yoctozeptohalf an hour seems extreme (-:14:59
fungiyeah, i agree. it wreaks havoc with our continuous deployment jobs, because if a server is hung then ansible tasks just pile up15:00
yoctozeptomeh :-(15:00
fungii want to say ansible itself will wait up to an hour for the ssh connection to complete, but we likely timed the playbook out before then in the job15:00
yoctozeptoguess so15:01
fungiif the playbook timeout were set higher, you'd have eventually seen ansible complain the host was unreachable, i think15:01
fungiafter, like, an hour15:01
fungiit's a pathological condition though, only seems to manifest if the server is reachable over the network and will negotiate an ssh key exchange but then never complete the login (extremely high system load, unresponsive rootfs, et cetera)15:03
fungiunfortunately we see those conditions far more often because... clouds15:03
yoctozeptocloudy vision15:21
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade gitea to 1.15.2
opendevreviewClark Boylan proposed opendev/system-config master: DNM force gitea failure for interaction
clarkbI think is mergeable from a gitea perspective now. The bug fixes we wanted have landed. However, it may be worth getting the opendev/assets work finished first so that we can stop relying on gitea for those assets as this change forces us to update their urls16:02
clarkbI'll put a hold on 800516 and we can double check the results16:02
clarkbthere is also a 1.14.7 which fixes an important bug with gitea dump which we don't use. I think we'll be ok without upgrading to 1.14.716:05
clarkbunfortunately they don't link to the issue and I'm having a really hard time finding what issue they fixed :/16:07
fungisounds worth checking then16:08
fungiyeesh, 1.6k issues16:09
fungiclarkb: maybe?16:09
fungioh, nevermind, wrong link16:10
clarkb that is the issue16:10
clarkbwhich is not linked to in the change log :/16:10
clarkbbut ya I don't think that is critical for us because we don't use gitea dump and we don't really have user accounts16:11
fungiyeah, looks like the pr which fixed that issue didn't include a reference to the issue number/url16:12
fungiand we're steering clear of 1.15.x releases for now, right?16:13
fungilooks like 1.15.1 was released yesterday16:13
clarkbfungi: yes the 1.15.x release is mergeable assuming 803231 passes testing and the held job on 800516 looks good (I just pushed those updates). But it changes the url for gerrit to load the opendev url in the theme. This is why the opendev/assets work is happening16:14
clarkbI think it might be better to finish up the opendev/assets work first if we can. But if necessary we can upgrade to 1.15.2 and restart gerrit and fix it up as required16:14
fungioh, right-o, forgot that was the driver for it16:14
clarkbBasically we're ready if our hand is forced, but the other path is a bit more agreeable16:15
clarkbOther things we should start thinking about are nodepool launcher restarts to pick up my fix for launcher reg cleanup in zk. And a general zuul restart if we can sneak it in around openstack release activity.16:16
clarkbI can do the nodepool launcher restarts later today. Right now my window for garden and bike ride is closing so I should do those things :)16:16
clarkbFor the zuul restart it might be good to get corvus' fix for the orphaned build request in first if that hasn't landed yet16:16
fungisure thing. i'm around, though working from the patio because the weather's conducive16:17
corvusoh it had a flake8 fail, i'll fix it up real q16:21
*** hjensas is now known as hjensas|afk16:29
*** ysandeep is now known as ysandeep|out16:41
*** jpena is now known as jpena|off16:41
*** odyssey4me is now known as Guest619818:03
clarkbhttps:// seems to render ok. I think that means we're still good with gitea 1.15.2 should we decide to go there before the assets updates19:31
clarkbI'm going to eat some lunch then do nodepool launcher restarts19:31
clarkbI guess while I'm looking at the launchers I should double check the inap resources got cleaned up19:32
*** dviroel is now known as dviroel|af19:35
*** dviroel|af is now known as dviroel|afk19:35
fungiyeah, that held node lgtm19:35
clarkbI've restarted the launcher on nl0220:17
clarkbif I see a few nodes properly transition state there I'll proceed to the others20:20
clarkbDetailed node error: MessagingTimeout <- the inmotion cloud seems unhappy I'm going to guess we lost the rabbitmq cluster based on that error and previous issues20:21
clarkbother nodes on nl02 are happy though, I'm going to proceed20:22
clarkbfungi: I think is ready to be approved if you want to take a look. nodepool list and nodepool image-list both lack inap entries20:26
clarkb#status log Restarted nl01-04 on 4edaeba70265396de415d2c7519b4ff8415e775020:31
opendevstatusclarkb: finished logging20:31
clarkbthanks for approving 80720520:43
opendevreviewGage Hugo proposed opendev/irc-meetings master: Move openstack-helm meeting to #openstack-helm
clarkbhuh I had missed that opensatck wanted to stop doing meetings in meeting channels20:49
opendevreviewMerged openstack/project-config master: Remove the inap provider from nodepool
clarkbYa checking /var/log/kolla/rabbitmq I see lots of disconnections20:53
clarkbthis seems similar to what happened last time. yuriys isn't here, but I wonder if we're still having nic issues there20:53

Generated by 2.17.2 by Marius Gedminas - find it at!