| *** tosky has quit IRC | 00:19 | |
| *** Meiyan has joined #opendev | 01:00 | |
| *** ysandeep|away is now known as ysandeep | 01:02 | |
| *** xiaolin has joined #opendev | 01:06 | |
| fungi | e-mail i promised about low-bandwidth mode workaround: http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015206.html | 01:55 |
|---|---|---|
| openstackgerrit | Merged openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 02:07 |
| *** xiaolin has quit IRC | 02:14 | |
| *** diablo_rojo has quit IRC | 03:11 | |
| *** ykarel|away is now known as ykarel | 04:08 | |
| *** diablo_rojo has joined #opendev | 04:38 | |
| *** priteau has joined #opendev | 04:50 | |
| AJaeger | config-core, please review https://review.opendev.org/730788 https://review.opendev.org/731989 https://review.opendev.org/732150 | 05:26 |
| openstackgerrit | Simon Westphahl proposed openstack/diskimage-builder master: Use kpartx option to update partition mappings https://review.opendev.org/728824 | 05:30 |
| *** priteau has quit IRC | 05:30 | |
| *** priteau has joined #opendev | 05:51 | |
| *** xiaolin has joined #opendev | 05:54 | |
| *** priteau has quit IRC | 05:55 | |
| *** gtema has joined #opendev | 05:57 | |
| *** priteau has joined #opendev | 06:01 | |
| *** priteau has quit IRC | 06:01 | |
| *** priteau has joined #opendev | 06:08 | |
| *** ysandeep is now known as ysandeep|brb | 06:21 | |
| *** sgw has quit IRC | 06:28 | |
| *** minmin has joined #opendev | 06:38 | |
| *** ravsingh has joined #opendev | 06:53 | |
| *** minmin has left #opendev | 06:55 | |
| *** ravsingh has quit IRC | 06:58 | |
| *** tkajinam has quit IRC | 07:13 | |
| *** tkajinam has joined #opendev | 07:13 | |
| *** ysandeep|brb is now known as ysandeep | 07:20 | |
| *** ysandeep is now known as ysandeep|lunch | 07:34 | |
| *** dpawlik has joined #opendev | 07:35 | |
| *** DSpider has joined #opendev | 07:38 | |
| *** dtantsur|afk is now known as dtantsur | 07:39 | |
| *** gtema has quit IRC | 07:44 | |
| *** rpittau|afk is now known as rpittau | 07:45 | |
| *** gtema has joined #opendev | 07:52 | |
| ianw | AJaeger/hrw: we've had some discussion on the wheel builds in the multi-arch ptg just now, see https://etherpad.opendev.org/p/Multi-arch-2020-VPTG | 07:54 |
| hrw | ianw: thanks, will read | 07:57 |
| jrosser | i am finished now with this held node | 0000000133 | openstack | opendev.org/openstack/openstack-ansible-lxc_hosts | openstack-ansible-lxc-dir-ubuntu-focal | .* | 1 | jrosser: investigate lxc errors on focal | 07:58 |
| jrosser | i learned all about how you can have a race condition with systemd-tmpfiles-setup.service cleaning out /tmp whilst ansible is using it as working space | 07:59 |
| *** moppy has quit IRC | 08:01 | |
| ianw | AJaeger: maybe you could look in on my rsync opts in "https://review.opendev.org/732478 [wip] make proper openafs publishing jobs" ... i want the rpmbuild/<RPMS/SRPMS> as an artifact, that we can then promote to tarballs.opendev.org | 08:01 |
| hrw | ianw: thx | 08:01 |
| *** moppy has joined #opendev | 08:01 | |
| ianw | jrosser: cool, have removed that | 08:01 |
| hrw | ianw: so progress is made, still stuff to do | 08:03 |
| ianw | hrw: in short, yes :) i think that fixing up the openafs builds for arm64 centos8 is a decent first step, and gives us flexibility to use that if we need. it would be even better to copy the wheels via the executor, but that will take some work | 08:05 |
| *** gtema has quit IRC | 08:05 | |
| hrw | ianw: is openafs part of c8? I can take a look at building it if needed | 08:06 |
| hrw | ianw: Debian has openafs 1.8.2 in repo so it should be easier | 08:07 |
| openstackgerrit | melissaml proposed openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/732490 | 08:10 |
| *** sshnaidm|afk is now known as sshnaidm | 08:21 | |
| AJaeger | ianw: rsync is always fun, the change LGTM but only testing will tell ;( | 08:21 |
| hrw | aarch64 jobs will probably timeout as we have a bunch of time occupying entries - can we salvage whatever was built in such case? | 08:25 |
| *** ykarel is now known as ykarel|lunch | 08:43 | |
| openstackgerrit | melissaml proposed openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/732490 | 08:44 |
| openstackgerrit | melissaml proposed openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/732490 | 08:46 |
| *** roman_g has joined #opendev | 08:50 | |
| *** tosky has joined #opendev | 08:57 | |
| *** stephenfin has joined #opendev | 08:58 | |
| *** stephenfin has quit IRC | 08:59 | |
| *** stephenfin has joined #opendev | 09:00 | |
| openstackgerrit | melissaml proposed openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/732490 | 09:02 |
| *** ysandeep|lunch is now known as ysandeep | 09:05 | |
| *** dirk has joined #opendev | 09:12 | |
| *** ykarel|lunch is now known as ykarel | 09:36 | |
| *** Meiyan has quit IRC | 09:44 | |
| *** diablo_rojo has quit IRC | 10:08 | |
| *** gtema has joined #opendev | 10:11 | |
| *** larainema has joined #opendev | 10:25 | |
| *** rpittau is now known as rpittau|bbl | 10:28 | |
| *** xiaolin has quit IRC | 10:44 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Enable linting of test-playbooks https://review.opendev.org/731471 | 11:27 |
| *** gtema has quit IRC | 11:30 | |
| *** hashar has joined #opendev | 11:51 | |
| *** gtema has joined #opendev | 12:04 | |
| zbr | clarkb: mordred: two basic e-r changes, https://review.opendev.org/#/c/731497/ (and parent), thanks. | 12:07 |
| *** panda|off is now known as panda | 12:08 | |
| *** rpittau|bbl is now known as rpittau | 12:21 | |
| *** ysandeep is now known as ysandeep|brb | 12:47 | |
| *** mlavalle has joined #opendev | 12:58 | |
| *** sgw has joined #opendev | 13:01 | |
| *** ysandeep|brb is now known as ysandeep | 13:03 | |
| johnsom | Hi there, I think ptgbot has an issue. It didn't update our room topics | 13:03 |
| AJaeger | infra-root ^ | 13:05 |
| *** ravsingh has joined #opendev | 13:23 | |
| *** avass has quit IRC | 13:26 | |
| *** rpittau is now known as rpittau|brb | 13:28 | |
| AJaeger | config-core, please review https://review.opendev.org/730788 https://review.opendev.org/731989 and https://review.opendev.org/732150 | 13:28 |
| *** panda has quit IRC | 13:39 | |
| *** panda has joined #opendev | 13:45 | |
| *** rpittau|brb is now known as rpittau | 14:01 | |
| *** hashar is now known as hasharAway | 14:05 | |
| *** mugsie_ is now known as mugsie | 14:06 | |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Update Octavia grafana dashboard https://review.opendev.org/732689 | 14:10 |
| *** hasharAway is now known as hashar | 14:11 | |
| *** gtema has quit IRC | 14:48 | |
| *** gtema has joined #opendev | 14:50 | |
| *** gtema_ has joined #opendev | 14:55 | |
| *** gtema has quit IRC | 14:57 | |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Update Octavia grafana dashboard https://review.opendev.org/732689 | 15:10 |
| *** rchurch has quit IRC | 15:16 | |
| *** gtema_ has quit IRC | 15:24 | |
| fungi | sslcheck is complaining that certs for graphite and the rax ord mirror are expiring in a month | 15:28 |
| fungi | looks like the cert for graphite was last updated april 3 | 15:29 |
| fungi | looks like acme.sh ran for it several says ago, but not since 2020-05-29 18:06:23z | 15:30 |
| clarkb | did we get a new cert or fail? | 15:31 |
| fungi | sampling some other servers, i see the same... we seem to have ceased updating le certs anywhere since friday | 15:32 |
| clarkb | if we have a new cert the apache workers may be stale | 15:32 |
| clarkb | oh I see | 15:32 |
| fungi | yeah, those are just the canary servers apparently | 15:32 |
| fungi | we're going to start seeing an increasing number of certs reported by sslcheck in the coming days if we don't fix it | 15:33 |
| fungi | anybody know of anything we might have merged between ~18:00-19:00 friday which could have affected it? | 15:34 |
| fungi | i'll look back through changes landing around then | 15:34 |
| fungi | er, i meant between ~18:00 friday and ~06:00 saturday | 15:34 |
| fungi | we seem to normally run this every 12 hours | 15:34 |
| clarkb | Im guessing fallout with mordreds reorg? | 15:41 |
| clarkb | we should check service le logs or mwybe the rename landed and that is related | 15:41 |
| clarkb | I'm finishing tea then can look more closely | 15:43 |
| *** ykarel is now known as ykarel|away | 15:43 | |
| fungi | i don't see anything merging in that window, and the stuff which merged right before it shouldn't have caused this | 15:49 |
| corvus | https://review.opendev.org/730991 hasn't merged yet | 15:50 |
| corvus | that's the one where we change service-letsencrypt to letsencrypt | 15:50 |
| fungi | https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-letsencrypt | 15:53 |
| fungi | we stopped running the job | 15:53 |
| corvus | it should still be scheduled to run in periodic | 15:54 |
| clarkb | ya I was just discovering that, its there in deploy and periodic | 15:55 |
| corvus | maybe there's a project config error preventing freezing | 15:55 |
| fungi | congress retirement related config errors in that tenant | 15:56 |
| clarkb | it has a soft dependency on infra-prod-base | 15:56 |
| clarkb | and that job also last ran friday | 15:56 |
| fungi | oh neat | 15:57 |
| corvus | https://zuul.opendev.org/t/openstack/builds?pipeline=periodic&project=opendev%2Fsystem-config | 15:57 |
| corvus | nothing has run in periodic for system-config since then | 15:57 |
| corvus | hrm, nothing logged about it | 16:01 |
| corvus | 2020-06-02 06:00:54,865 DEBUG zuul.Pipeline.zuul.periodic: [e: 3762651acb2547e3aa5ad576f3ce6f6c] Freezing job graph for <QueueItem 0x7f63f93c1d90 for <Branch 0x7f63f958f6d0 opendev/system-config refs/heads/master updated None..None> in periodic> | 16:01 |
| corvus | 2020-06-02 06:00:54,865 DEBUG zuul.Pipeline.zuul.periodic: [e: 3762651acb2547e3aa5ad576f3ce6f6c] Reporting change <Branch 0x7f63f958f6d0 opendev/system-config refs/heads/master updated None..None> | 16:01 |
| corvus | just <shrug> didn't do anything | 16:01 |
| fungi | yet we're still running hourly jobs from opendev-prod-hourly | 16:02 |
| clarkb | as a double check only the reorg or roles has landed from mordreds stack | 16:04 |
| clarkb | none of the job changes have gone in yet I think | 16:05 |
| mordred | what did I do? | 16:05 |
| fungi | mordred: we're sure you did something, just give us time to fabricate the evidence ;) | 16:05 |
| clarkb | mordred: probably nothing. Our periodic jobs for system-config last ran friday. I'm confirming that your reorg stack hasn't gotten to the point where it changes job stuff | 16:05 |
| corvus | yeah, and i can't see that e8716e742e5da8cdde45ff71d4521c25f04f7397 could have changed this | 16:05 |
| mordred | clarkb: yeah - we haven't gotten far enough along in that stack | 16:06 |
| mordred | to like, do things with jobs | 16:07 |
| corvus | oh wait, there may be more debug msgs | 16:09 |
| corvus | Exception: Job infra-prod-run-accessbot depends on infra-prod-service-eavesdrop which was not run. | 16:10 |
| corvus | there it is | 16:10 |
| * clarkb finally finds the logs, they are in debug.log.1.gz | 16:11 | |
| fungi | huh, i thought we only ran infra-prod-run-accessbot in deploy | 16:11 |
| fungi | so i guess this was related to the move of infra-prod-service-eavesdrop from periodic to hourly | 16:12 |
| corvus | likely this change: https://review.opendev.org/726379 | 16:12 |
| clarkb | fungi: ya I guess we need to move all of the things that depends on service-eavesdrop to hourly | 16:12 |
| fungi | ahh, yep, infra-prod-run-accessbot runs in periodic without declaring a dependency on infra-prod-service-eavesdrop, but runs in deploy declaring a dependency on it | 16:13 |
| fungi | is the dependencies list transitive to other pipelines where it's not declared? | 16:14 |
| corvus | well, it has a dependency on it in the job definition | 16:14 |
| fungi | ahh | 16:14 |
| fungi | so it just happened to work in periodic because they were both in there | 16:15 |
| clarkb | the deploy dependency seems to be there to make it soft | 16:15 |
| corvus | yep | 16:15 |
| fungi | okay, so here's the dilemma: we want to run infra-prod-service-eavesdrop frequently so we can deploy software to the server more rapidly, we want to run infra-prod-run-accessbot infrequently because freenode come in here and ask if something is wrong when we're updating modelock on hundreds of channels | 16:17 |
| fungi | we don't want to run them in deploy because we're planning to split stuff between different tenants | 16:18 |
| corvus | i think ptgbot is in the wrong tenant | 16:18 |
| fungi | yeah, or ptgbot needs to be deployed with a totally separate job | 16:19 |
| fungi | right now we have several distinct services getting deployed to eavesdrop from a single playbook | 16:19 |
| fungi | or at least from a single job | 16:20 |
| corvus | i guess we want to keep periodic and hourly mutually exclusive? | 16:20 |
| fungi | maybe? i agree putting it in both could be a possible workaround | 16:21 |
| corvus | because once a day, they run at the same time? | 16:21 |
| corvus | so if we did do that, we'd need a mutex, and we've have a possible state race? | 16:21 |
| fungi | i guess we decided deploy fires infrequently enough that doing jobs in deploy and hourly don't race enough to matter? | 16:22 |
| clarkb | weren't we going to make accessbot only run if there were changes? | 16:22 |
| clarkb | then we can run it hourly and let the ansible skip running it if there is no delta? | 16:23 |
| clarkb | but ya we could also split the eavesdrop job into more of its constituent parts and remove the hard deps between them unless necessary | 16:23 |
| clarkb | (I think that would mean ptgbot could run entirely independently) | 16:23 |
| corvus | oh, i think we should run accessbot fully at least daily | 16:25 |
| clarkb | ah mabe that is why we didn't go with that plan (I seemed to recall that was the first suggestion for fixing the freenode concerns) | 16:26 |
| corvus | it's sort of designed as an enforcer; if channel settings drift too much, i think we'd want to catch that earlier rather than later | 16:26 |
| clarkb | got it | 16:26 |
| *** rpittau is now known as rpittau|afk | 16:32 | |
| *** hashar has quit IRC | 16:33 | |
| *** hashar has joined #opendev | 16:33 | |
| fungi | yeah, running daily and also when changes merge for its config or script | 16:33 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Move run-eavesdrop from periodic-hourly to periodic https://review.opendev.org/732737 | 16:34 |
| *** diablo_rojo has joined #opendev | 16:34 | |
| corvus | fungi, clarkb, mordred: ^ i think that's our minimum -- but it also seems like running in both hourly and daily could be an option? | 16:34 |
| clarkb | corvus: the issue with running accessbot hourly is freenode didn't seem to appreciate that? | 16:35 |
| fungi | well, it got them worried since they saw us constantly enforcing modelocks on all those channels | 16:37 |
| corvus | clarkb: right, i'm not suggesting we run accessbot hourly, only eavesdrop | 16:37 |
| corvus | but yes, that's why we don't just have them both in hourly | 16:37 |
| corvus | so our choices right now (without restructuring the jobs to split out ptgbot or something similar) are: a) eavesdrop+accessbot in daily; b) eavesdrop+accessbot in daily, eavesdrop in hourly | 16:39 |
| clarkb | oh I see I had B the wrong way around in my head | 16:39 |
| clarkb | both in hourly and both in daily | 16:39 |
| *** dtantsur is now known as dtantsur|afk | 16:41 | |
| *** ravsingh has quit IRC | 16:43 | |
| *** roman_g has quit IRC | 16:44 | |
| corvus | yeah, that's the thing we can't do because we can't have accessbot in hourly due to freenode | 16:45 |
| *** ysandeep is now known as ysandeep|away | 17:05 | |
| openstackgerrit | Merged opendev/system-config master: Move run-eavesdrop from periodic-hourly to periodic https://review.opendev.org/732737 | 17:12 |
| *** tkajinam has quit IRC | 17:16 | |
| *** dpawlik has quit IRC | 17:20 | |
| *** hashar has quit IRC | 17:25 | |
| *** dpawlik has joined #opendev | 17:29 | |
| *** dpawlik has quit IRC | 17:33 | |
| *** dpawlik has joined #opendev | 17:33 | |
| AJaeger | config-core, please review https://review.opendev.org/730788 https://review.opendev.org/731989 and https://review.opendev.org/732150 | 17:39 |
| *** priteau has quit IRC | 18:09 | |
| openstackgerrit | Hervé Beraud proposed openstack/diskimage-builder master: Stop to use the __future__ module. https://review.opendev.org/732833 | 18:11 |
| *** priteau has joined #opendev | 18:37 | |
| openstackgerrit | Hervé Beraud proposed openstack/project-config master: Stop to use the __future__ module. https://review.opendev.org/732911 | 18:45 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add addons args in ensure-kubernetes role https://review.opendev.org/731523 | 19:49 |
| fungi | i'm going to disappear for a bit and try to get in a long nap before 04:00z | 20:05 |
| openstackgerrit | Stephen Finucane proposed opendev/bindep master: Handle non-en_US locales https://review.opendev.org/732992 | 20:05 |
| openstackgerrit | Stephen Finucane proposed opendev/bindep master: Switch to stestr https://review.opendev.org/732994 | 20:08 |
| openstackgerrit | Tim Burke proposed openstack/project-config master: Prepare to retire swift-specs https://review.opendev.org/732999 | 20:50 |
| *** priteau has quit IRC | 20:55 | |
| *** priteau has joined #opendev | 20:56 | |
| openstackgerrit | Tim Burke proposed openstack/project-config master: Retire swift-specs https://review.opendev.org/733003 | 20:56 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support https://review.opendev.org/720302 | 21:04 |
| corvus | mordred: ^ dusting that off to see where we are now that all the changes have landed | 21:05 |
| *** priteau has quit IRC | 21:32 | |
| *** gtema has joined #opendev | 22:06 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add addons args in ensure-kubernetes role https://review.opendev.org/731523 | 22:11 |
| *** gtema has quit IRC | 22:14 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Indicate firefox is suboptimal in meetpad https://review.opendev.org/733022 | 22:44 |
| corvus | fungi, clarkb: ^ based on what we've learned so far, these might be good things to set. | 22:45 |
| *** tkajinam has joined #opendev | 22:50 | |
| openstackgerrit | Merged openstack/project-config master: Update Octavia grafana dashboard https://review.opendev.org/732689 | 23:08 |
| *** tosky has quit IRC | 23:34 | |
| mordred | corvus: it failed on apt-get update on kubic with a cert issue - I think it might want a recheck? | 23:42 |
| corvus | mordred: i'll do that and check back in on it tomorrow :) | 23:43 |
| mordred | corvus: cool | 23:43 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!