Friday, 2020-09-18

fungiokay, 752584 is a wip patch to undo the texlive-full addition, and 752585 is a dnm change to exercise it from nova's docs build00:01
ianwclarkb: to be clear; /afs/.openstack.org/mirror/deb-docker will be left with xenial/bionic/focal top-level directories only00:01
clarkbyup, I was just curious how jobs were functional before00:02
clarkband its the old stale stuff they rely on00:02
openstackgerritMerged opendev/base-jobs master: Fix docker base location  https://review.opendev.org/74627100:07
clarkbwoo nodepool image promote job is running now00:11
clarkbthat should get us an up to date dib instal lwhich should allow us to build tumbleweed images00:11
clarkbthen we can delete nb04 :)00:11
fungiand then i can close out the rax-dfw cinder maintenance ticket00:12
clarkbcloud launcher is running now then its nodeool which should grab the new image00:16
* clarkb is impatient00:16
*** ricolin has quit IRC00:22
*** hamalq has quit IRC00:30
*** kden has joined #opendev00:32
gmannfungi: updated tested patch with right doc job -https://review.opendev.org/#/c/752585/00:34
openstackgerritMerged opendev/system-config master: Update deb-docker path  https://review.opendev.org/74627200:55
fungithanks gmann!01:03
ianw#status log cleared old docker repo from /afs/.openstack.org/mirror/deb-docker ; jobs should now be using repo-specific dirs01:03
openstackstatusianw: finished logging01:04
openstackgerritMerged openstack/project-config master: Remove noop jobs for python-adjutantclient  https://review.opendev.org/75099001:05
fungigmann: clarkb: looks like it's probably wanting an additional font package: https://zuul.opendev.org/t/openstack/build/e2c9a48411c7416e891c65c2cb96f92a/log/sphinx-build-pdf.log#6159-616801:09
fungii'm hitting the hay though01:11
*** zbr7 has joined #opendev01:16
*** zbr has quit IRC01:19
*** zbr7 is now known as zbr01:19
ianwE: Failed to fetch https://mirror.regionone.linaro-us.opendev.org/ubuntu-ports/dists/bionic/main/binary-arm64/Packages  403  Forbidden [IP: 2604:1380:4111:3e54:f816:3eff:fe17:6b17 443]01:20
ianwok, back to this arm mirror01:20
ianwit has oopsed01:24
ianwsame as https://lists.openafs.org/pipermail/openafs-info/2020-August/042959.html01:25
ianwi'm going to clear the cache and reboot it01:25
openstackgerritMerged opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644901:36
openstackgerritMerged opendev/system-config master: Ansible devel testing: install ansible-collections from checkout  https://review.opendev.org/74759701:36
*** hamalq has joined #opendev01:37
*** hamalq_ has joined #opendev01:41
johnsomThat font is in the texlive-base package01:42
*** hamalq has quit IRC01:44
*** hamalq_ has quit IRC03:18
*** ricolin has joined #opendev03:24
*** kden has quit IRC03:44
*** kden has joined #opendev03:47
*** ykarel|away has joined #opendev04:01
openstackgerritMerged opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org  https://review.opendev.org/74781304:46
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Rename duplicating 10-debian-minimal.bash  https://review.opendev.org/75245604:53
*** marios has joined #opendev04:57
*** ykarel|away is now known as ykarel05:00
openstackgerritMerged openstack/project-config master: Remove openstack-python-jobs from GBP templates  https://review.opendev.org/75257505:05
*** marios has quit IRC05:11
*** marios has joined #opendev05:13
*** hamalq has joined #opendev05:44
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed zuul/zuul-jobs master: Add support to use stow for ensure-python  https://review.opendev.org/75161105:56
*** DSpider has joined #opendev05:57
*** ysandeep|away is now known as ysandeep06:07
*** Tengu has quit IRC06:07
*** slaweq has joined #opendev06:14
*** qchris has quit IRC06:21
*** auristor has quit IRC06:21
openstackgerritlikui proposed openstack/diskimage-builder master: replace imp module  https://review.opendev.org/75123606:25
*** auristor has joined #opendev06:28
*** hamalq has quit IRC06:29
*** hamalq has joined #opendev06:32
*** DSpider has quit IRC06:32
*** qchris has joined #opendev06:33
*** hamalq has quit IRC06:44
*** zbr5 has joined #opendev06:45
*** zbr5 has quit IRC06:49
*** zbr has quit IRC06:49
*** hamalq has joined #opendev06:52
*** seongsoocho has quit IRC06:53
*** ShadowJonathan has quit IRC06:53
*** mwhahaha has quit IRC06:53
*** johnsom has quit IRC06:53
*** Open10K8S has quit IRC06:53
*** rpittau|afk has quit IRC06:53
*** donnyd has quit IRC06:53
*** rm_work has quit IRC06:53
*** TheJulia has quit IRC06:53
*** ildikov has quit IRC06:53
*** hillpd has quit IRC06:54
openstackgerritzbr proposed zuul/zuul-jobs master: WIP: Update ensure-docker for new release  https://review.opendev.org/75263006:54
*** Open10K8S has joined #opendev06:55
*** zbr has joined #opendev06:55
*** hillpd has joined #opendev06:55
*** ShadowJonathan has joined #opendev06:55
*** mwhahaha has joined #opendev06:55
*** ildikov has joined #opendev06:55
*** seongsoocho has joined #opendev06:55
*** hashar has joined #opendev06:56
*** rpittau|afk has joined #opendev06:56
*** donnyd has joined #opendev06:57
*** TheJulia has joined #opendev06:57
*** johnsom has joined #opendev06:57
*** andrewbonney has joined #opendev07:01
*** hamalq has quit IRC07:05
*** rm_work has joined #opendev07:08
openstackgerritMerged opendev/elastic-recheck master: Enable configuration via environment variables  https://review.opendev.org/74689007:14
*** tosky has joined #opendev07:25
*** DSpider has joined #opendev07:50
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
openstackgerritMerged openstack/diskimage-builder master: Rename duplicating 10-debian-minimal.bash  https://review.opendev.org/75245608:01
*** ykarel_ has joined #opendev08:02
*** ykarel has quit IRC08:05
*** ykarel_ is now known as ykarel08:10
ianwclarkb: note dib 3.3.1 released with a fix for debian builds08:18
ianwclarkb: https://review.opendev.org/#/c/742057/ might be one to look at that would rebuild the image too, uses focal builds of tools08:19
*** dtantsur|afk is now known as dtantsur08:27
openstackgerritzbr proposed zuul/zuul-jobs master: Update ensure-docker for new releases  https://review.opendev.org/75263008:43
openstackgerritlikui proposed openstack/diskimage-builder master: replace imp module  https://review.opendev.org/75123608:45
*** mordred has quit IRC08:52
*** Eighth_Doctor has quit IRC08:53
*** kden has quit IRC08:57
*** Eighth_Doctor has joined #opendev09:18
*** Eighth_Doctor has quit IRC09:19
*** mordred has joined #opendev09:26
*** Eighth_Doctor has joined #opendev09:55
*** hashar is now known as hasharAway10:47
*** andrewbonney has quit IRC10:51
*** ysandeep is now known as ysandeep|afk11:01
*** ysandeep|afk is now known as ysandeep11:42
*** hasharAway is now known as hashar11:51
openstackgerritMerged opendev/elastic-recheck master: Replace testr with pytest  https://review.opendev.org/75248412:13
*** marios is now known as marios|call12:32
*** andrewbonney has joined #opendev12:36
*** marios|call is now known as marios12:43
fungijohnsom: interesting then, because texlive-base should already be getting installed (it's marked as a dependency for texlive-xetex which we install, for example)12:53
fungigerritlib reviewers: could use a second set of eyes on https://review.opendev.org/750849 now that it's confirmed to be helpful for keeping the gerrit event stream stable on some networks13:08
*** owalsh has quit IRC13:11
zbrfungi: i can have an look on gerritlib13:15
*** ysandeep is now known as ysandeep|session13:16
fungithanks! i think the same author was also planning to push a corresponding gerritbot change to make it possible to turn on ssh keepalives via configuration13:16
fungithough i don't think i've seen that one yet13:16
fungibut that's basically where the reported problem surfaced13:17
fungi(gerritbot's event stream just going dead when there was a lull in activity)13:17
fungilikely a firewall or nat with aggressive idle state timeout policy13:18
*** tosky_ has joined #opendev13:18
*** tosky is now known as Guest8304713:18
*** tosky_ is now known as tosky13:18
zbrfungi: i already had one change in that area and i am aware it gets stuck on error instead of ending the thread.13:22
zbrfungi: maybe you can comment on https://review.opendev.org/#/c/729966/5/gerritlib/gerrit.py13:27
zbri personally do not think adding a condition for non linux would be the best approach on it13:28
zbrIMHO "event & select.POLLIN" is the correct way to check for incoming data, not the equal13:29
zbrcurrent implementation crashes on OOB messages, if it would be a car is like stopping the car because an unknown road sign was detected.13:30
zbri did read the spec and there is nowere written that an app must stop processing when POLLPRI appears13:31
chandankumarweshay|ruck: https://review.opendev.org/#/c/752685/13:43
chandankumarsorry13:43
*** kden has joined #opendev13:50
openstackgerritzbr proposed opendev/gerritlib master: Avoid crash due to POLLIN event check  https://review.opendev.org/72996613:57
openstackgerritzbr proposed opendev/elastic-recheck master: WIP: Create elastic-recheck container image  https://review.opendev.org/72962314:09
zbrfungi: does the new IS_LINUX approach seems ok to you?14:11
*** hashar has quit IRC14:12
*** hashar has joined #opendev14:12
zbri still failed to find a single place that requires user to act on POLLPRI, regardless of the platform. The fact that this flag is used should not crash an implementation.14:12
fungizbr: yep, started looking at it... if !IS_LINUX is always assumed to be darwin then that's probably okay, i guess there's not a more exact way to match macosx/darwin kernels?14:13
zbrI can do == Darwin, but this would put Java in the linux box.14:13
zbrbut I am not aware of any jython users.14:14
zbrif it breaks for them, i am happy to review a patch14:14
zbrthe reality is that at least in this case, there is not something weird done by Darwin, it is still within the poll specs.14:14
zbrand i lost hope that paramiko will implement that bit14:15
zbrfungi: btw, are we ok to drop support for <py36 on gerritlib?14:19
*** ysandeep|session is now known as ysandeep14:19
fungizbr: yeah, we can probably drop it. i *think* we're only running it in docker containers for opendev now14:20
zbrnot yet, but soon will be container only.14:20
*** ysandeep is now known as ysandeep|away14:21
zbri have the e-r container build but not ready. anyway, e-r is already using py36.14:21
fungias for pollpri, packet captures seemed to suggest that darwin (this may be specific to their mach implementation, or might be an inherited bsdism) sets it when doing psh/urg14:21
openstackgerritzbr proposed opendev/gerritlib master: Allow getPlugins to accept a format  https://review.opendev.org/23815914:22
fungioh! right, e-r uses gerritlib... we might want to hold off merging the >=3.6 change until we're deployed on the containerized version. we just recently shot ourselves in the feet dropping support for older python in storyboard while we were not yet ready to switch to running from containers14:23
*** owalsh has joined #opendev14:29
*** dmellado has quit IRC14:40
*** dmellado_ has joined #opendev14:40
*** dmellado_ is now known as dmellado14:41
clarkbI dont have any rush to drop python3.5 or 2.7 support in gerritlib14:50
*** ykarel is now known as ykarel|away14:59
zbrclarkb: afaik, we already use py36 on e-r, right? i seen the puppet change merged long time ago.15:19
zbrer codebase already requires py36, so i would not count it as a reason to keep backwards copatibility on gerritlib.15:21
clarkbjeepyb still uses it too15:24
clarkbif wecan audit all the use and be faily sure we dont need 35 or 27 then wecan go ahead15:25
zbrclarkb: https://opendev.org/opendev/jeepyb/src/branch/master/tox.ini#L815:25
zbrclarkb: do we have any chance to enable periodic-weekly today in order to get an idea about broken jobs by monday?15:26
clarkbzbr: that is a zuul question and I didnt see any opposition when I a sked #zuul so ya I think we can land it15:27
clarkb(we should keep questions like that in #zuul)15:27
zbri tried, not much interest on zuul-jobs, especially in that (testing) area.15:27
clarkbre that tox config item that includes 3515:28
clarkbso I guessits mostly checking if 35 is used15:28
clarkbzbr its off topic for opendev though15:28
*** mlavalle has joined #opendev15:41
*** tkajinam has quit IRC15:44
*** marios has quit IRC15:47
*** ykarel|away has quit IRC15:49
*** dtantsur is now known as dtantsur|afk16:10
*** hashar has quit IRC16:21
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Added message broker (RabbitMQ) settings  https://review.opendev.org/75272916:43
clarkbnb02 did successfully build a tumbleweed image16:47
clarkbit uploaded so I've deleted the nb04 image16:48
clarkbI think we'll be able to delete nb04 shortly16:48
*** Tengu has joined #opendev16:54
*** kden has quit IRC17:01
*** hamalq has joined #opendev17:01
*** iurygregory has quit IRC17:10
*** andrewbonney has quit IRC17:11
clarkbfungi: for that puppet lint error it originates in puppet 3 vendored safe_yaml. We run puppet 4 now though so maybe we can fix that with a change of the puppet version we lint with17:16
clarkbhttps://tickets.puppetlabs.com/browse/PUP-3796 suggests another fix is to intsall a newer safe_yaml17:17
fungioh! thanks, and yes i was down the rabbit hole of vendored puppet libs17:18
fungiclarkb: this is where we'd bump the puppet version? https://opendev.org/opendev/puppet-openstack_infra_spec_helper/src/branch/master/puppet-openstack_infra_spec_helper.gemspec#L2417:24
clarkbfungi: ya that looks right, I think the ~> means match 3.8.anything but not 3.9 or 4.x17:24
clarkbnot sure what othe rfallout it will have there but we don't puppet3 anymore so is an improvment17:25
fungiwe seem to be installing puppetlabs-release-pc1 1.1.0-4xenial currently17:26
fungiahh, and puppet-agent 1.10.14-1xenial which corresponds to puppet 4.10.1217:27
fungiso make that ~>4.10 i suppose17:27
openstackgerritJeremy Stanley proposed opendev/puppet-openstack_infra_spec_helper master: Match Puppet version used on servers  https://review.opendev.org/75273417:30
fungiinfra-puppet-core: ^17:31
clarkbfungi: we should probably dependson to ensureit works?17:32
fungiclarkb: that was my plan, yeah17:32
fungiwas just gonna edit the commit message on 752729 accordingly17:33
openstackgerritThomas Bachman proposed openstack/project-config master: Remove legacy gate jobs  https://review.opendev.org/75273517:46
fungiclarkb: hrm... looks like the unit test didn't like it17:51
fungineed json_pure (~> 1.8) and okogiri (~> 1.5)17:53
sgwMorning folks, I am working on setting up a new starlingx repo in project-config, I have a question about importing an existing repo vs start with empty one.  I know it will import all tags, branches and history.  So we should "clean" it up first before setting the upstream, correct?17:57
openstackgerritJeremy Stanley proposed opendev/puppet-openstack_infra_spec_helper master: Match Puppet version used on servers  https://review.opendev.org/75273417:58
fungisgw: that's correct. one way to do it is to push a cleaned-up fork of it somewhere with only the branches/tags/files/history you want us to import if you don't want to clean up the original17:59
sgwfungi: thanks, that's what I thought based on reading the creators doc.17:59
sgwI expect we will work on this on Monday.  I have it ready to get reviewed once we have a good repo to pull from.18:00
fungigood luck! don't hesitate to ask any other questions you might have18:01
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Added message broker (RabbitMQ) settings  https://review.opendev.org/75272918:08
clarkbfungi: ya not surprising thatp uppet 4 conflicts a bit with our old pins18:08
fungithat seems to have satisfied it, now for the depends-on18:08
openstackgerritClark Boylan proposed opendev/system-config master: Remove nb04  https://review.opendev.org/75274018:12
openstackgerritClark Boylan proposed openstack/project-config master: Remove old nodepool builder configs  https://review.opendev.org/75274118:14
clarkbinfra-root I think we are about as ready as we can be to do ^. Once those land I can do the manual steps to rm the server and its volume18:14
fungiclarkb: https://a77154d15efa6d7129c5-42788075eb1272bb581787feec9a8f1b.ssl.cf1.rackcdn.com/752729/2/check/legacy-puppet-lint/5d56d7e/job-output.txt that was rerun with the dep on the puppet upgrade, looks like it still ended up fetching puppet 3.8.7... oh! i bet we can't depends-on in that legacy job18:20
clarkboh ya that could be18:20
clarkbshould we just go for it and revert if it makes it worse?18:21
fungimmm, no dice, log also says:18:21
fungiopendev.org/opendev/puppet-openstack_infra_spec_helper checked out to: 619783f68369d954e9c7029d144b1ea5ebc6afc1 Match Puppet version used on servers18:21
funginowait18:22
clarkbUsing puppet-openstack_infra_spec_helper 0.0.1 from https://git.openstack.org/openstack-infra/puppet-openstack_infra_spec_helper (at master@1e6387b)18:22
fungiFetching https://git.openstack.org/openstack-infra/puppet-openstack_infra_spec_helper18:22
fungiyeah18:22
clarkblets merge it, its testing only with no impact on prod18:22
fungiokay, so yeah regardless we're not testing with the version of puppet we're using right now, we should forge ahead i think18:22
clarkbif it doesn't work we roll forward or revert18:22
fungiinfra-puppet-core: any disagreements?18:23
clarkbI've +2'd it if you want to approve18:23
fungiyeah, just waiting for my gertty vm to stop swap-thrashing for a sec18:25
clarkboh I can do it18:26
fungiwell, it also provides a convenient excuse to wait a few minutes for objections18:29
fungibut easy enough to revert, as you say18:29
clarkbre memory. my laptop only has 8GB because its a single "dimm" soldered setup with intel memory controller that can only do 8GB per dimm18:31
clarkbrecent firefox has been particularly bad at consuming all the memory18:31
clarkbdmsimard: the fedora-31 issues have me noticing that fedora-30 is still in use by ara as well. Any chance we can switch that to fedora-32?18:33
clarkbalso looks like our fedora-latest nodeset is set to fedora-30 too18:34
clarkbianw: for when your next week starts should we think about bumping fedora-latest to fedora-32 then cleaning up nodepool and ara and dib's use of fedora-30?18:35
clarkbthen we can remove the fedora-30 image from nodepool entirely18:37
clarkbalso not to completely derail fedora work but I wonder if centos stream would be a better fit for us18:37
fungior fedora-latest and let folks deal with the constant churn rather than needing us to do it for them18:39
openstackgerritMerged opendev/puppet-openstack_infra_spec_helper master: Match Puppet version used on servers  https://review.opendev.org/75273418:40
openstackgerritAndreas Jaeger proposed opendev/base-jobs master: Switch fedora-latest to version 32  https://review.opendev.org/75274418:40
AJaegerianw, clarkb, fungi ^18:40
clarkbAJaeger: thanks18:41
fungiaha!18:41
fungii thought i remembered that being a thing18:42
clarkbya it is, the problem is that anyone can use any valid label18:42
clarkbthey don't have to use the fedora-latest nodeset18:42
clarkbcentos stream sort of forces that issue :)18:42
openstackgerritAndreas Jaeger proposed openstack/project-config master: Switch grafana nodepool to Fedora 32  https://review.opendev.org/75274518:42
AJaegerand one more ^18:42
clarkbbut I'm not quite sure how well centos stream maps onto what people would like to test with fedora18:43
clarkbis it just keeping up with the rhel pipeline? or do people actually run fedora in production? etc18:43
clarkbAJaeger: for https://review.opendev.org/752744 I'm thinking we should announce that18:43
clarkbAJaeger: maybe we can send an email today and plan for it to chagne next week?18:43
AJaegerclarkb: works for me18:44
AJaegerwill you send it, please?18:44
clarkbyes I can send it18:45
AJaegerthanks18:46
AJaegerI can WIP the change - until when?18:46
AJaegerTuesday?18:46
clarkbya that works, that is the 22nd18:47
clarkbI'll note that in my email18:47
AJaegerok18:47
clarkbthere are a couple repos we have to update that use the label directly which may mean the actual removal happens later but for -latest tuesday sounds good18:47
AJaegerclarkb: just 2 more, isn't? nodepool and ara-infra?18:49
dmsimardo/18:49
clarkbAJaeger: and dib itself18:49
dmsimardgotta check for fedora-30, just upgraded from 31 to 32 and didn't realize there might have been a fedora-30 around still18:50
dmsimardah ok, found it in ara-infra18:50
clarkbI'm trying to pull on all the nodepool cleanup threads while I have all this stuff paged in :)18:51
clarkbemail sent to service-announce18:51
fungithanks!18:52
dmsimardgot a patch out to update ara-infra to f32, will land it asap18:53
clarkbdmsimard: thanks! its probably not today urgent but next would would be great18:53
clarkbfungi: puppet-lint is running now19:02
clarkbnext we see if it is happier19:02
fungiyup, i have the console stream up19:02
fungifetching puppet 4.10.12!19:08
fungilooks like it got past the install phase for everything19:08
fungiokay, that worked. great find on the puppet version being too low19:10
dmsimardclarkb: fyi ara-infra has moved to f3219:23
dmsimardmerged just now19:23
clarkbthabks!19:24
fungiheading out to run some quick errands, brb19:34
openstackgerritMerged opendev/puppet-openstackid master: Added message broker (RabbitMQ) settings  https://review.opendev.org/75272919:39
*** slaweq has quit IRC19:41
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: wip: ensure-zookeeper: add role to setup zookeeper  https://review.opendev.org/75275719:52
clarkbfungi: it looks like the deploy for https://review.opendev.org/752729 failed after the LE job failed20:21
clarkblooking at the LE logs I'm not sure why it failed though20:21
clarkbactually no the job I'm looking at was from ~1800UTC20:24
fungiyeah, the puppet stuff runs in the hourly pipeline and logs to /var/log/ansible/remote_puppet_else.yaml.log on bridge20:37
fungilooks like the openstackid-dev manifest was applied successfully20:37
clarkbthe LE failure happened in the deploy pipeline not the hourly pipeline20:37
clarkband lokoing at it I really have no idea why20:38
fungipossibly related, graphite02 has been causing sslcheck errors for a week or so, but looking into it has been a low priority because it's not in production yet20:38
*** owalsh has quit IRC20:46
*** owalsh has joined #opendev21:14
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: wip: ensure-zookeeper: add role to setup zookeeper  https://review.opendev.org/75275721:15
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: wip: ensure-zookeeper: add role to setup zookeeper  https://review.opendev.org/75275721:42
corvusM6.8 central mid atlantic ridge -- don't see activity there very often21:58
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: ensure-zookeeper: add role to setup zookeeper  https://review.opendev.org/75275722:11
clarkbnodepool has updated and should successfully build debuntu images again22:27
fungicorvus: yikes, on the lookout for tsunami here in that case22:34
* fungi only half serious22:34
fungiusgs also apparently recorded a 5.7 there early utc yesterday22:37
fungiand a 6.6 a couple weeks ago22:37
*** mlavalle has quit IRC22:38
*** DSpider has quit IRC22:50
*** hamalq has quit IRC23:02
openstackgerritClark Boylan proposed zuul/zuul-jobs master: ensure-zookeeper: add role to setup zookeeper  https://review.opendev.org/75275723:27
*** tosky has quit IRC23:31
*** hamalq has joined #opendev23:48

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!