| *** michael-beaver has quit IRC | 00:32 | |
| *** dchen has quit IRC | 00:35 | |
| *** dchen has joined #openstack-infra | 00:36 | |
| *** efried has joined #openstack-infra | 00:43 | |
| *** zhurong has joined #openstack-infra | 00:59 | |
| *** ricolin has joined #openstack-infra | 01:04 | |
| *** imacdonn has quit IRC | 01:10 | |
| *** imacdonn has joined #openstack-infra | 01:11 | |
| *** corvus has quit IRC | 01:12 | |
| *** corvus has joined #openstack-infra | 01:12 | |
| *** bobh has joined #openstack-infra | 01:17 | |
| *** bobh has quit IRC | 01:22 | |
| * ianw googles Bonaire | 01:27 | |
| ianw | looks great! :) | 01:27 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 01:48 | |
| *** rfarr has quit IRC | 02:03 | |
| *** rfarr has joined #openstack-infra | 02:04 | |
| *** tinwood has quit IRC | 02:08 | |
| *** tinwood has joined #openstack-infra | 02:10 | |
| openstackgerrit | Merged opendev/system-config master: Add OVH GRA1 mirror https://review.opendev.org/667543 | 02:15 |
| *** auristor has joined #openstack-infra | 02:43 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** yamamoto has joined #openstack-infra | 02:44 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:53 | |
| *** auristor has quit IRC | 02:55 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:56 | |
| *** bhavikdbavishi has quit IRC | 02:58 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:58 | |
| *** auristor has joined #openstack-infra | 03:10 | |
| *** bhavikdbavishi has quit IRC | 03:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt-install-txt-record: skip disabled hosts https://review.opendev.org/668344 | 03:10 |
| *** bhavikdbavishi has joined #openstack-infra | 03:11 | |
| auristor | ianw: 5.2-rc7, tagged yesterday, includes the kafs fixes in case you might want to update the kernel on iad | 03:24 |
| ianw | auristor: thanks, yep that was what i was going to roll out to the new gra1 server before i found a small issue getting certificates for it above ^^ :) | 03:27 |
| auristor | sorry, I've been offline all weekend | 03:28 |
| ianw | me too :) | 03:28 |
| ianw | i don't think we're having any issues with kafs as such. we can investigate the offline messages but they don't really seem to cause an issue i can see | 03:28 |
| ianw | the hard crashes seem to be filecache related, although it's survived all weekend (though it's much quieter) | 03:29 |
| auristor | the offline messages are the result of the volumes being taken offline for the release. | 03:29 |
| auristor | I started to review the retry logic with dhowells on Friday but we ran out of time. | 03:29 |
| auristor | The reason I'm interested in reviewing the audit logs from the fileservers is that I'm trying to figure out of the RO on afs01 is being taken offline during the release or if its just the RO on afs02 that is offline. | 03:31 |
| auristor | That will tell us whether kafs is failing to failover to afs01 from afs02. | 03:32 |
| auristor | It might be a case of timing. afs01 takes the RO offline to update its clone from the RW. kafs sees that and marks the volume instance offline. It then switches to afs02 but by the time it does, the RO on afs02 has been taken offline. | 03:33 |
| auristor | I will continue to review the retry logic this week | 03:36 |
| ianw | auristor: so basically, the same way i set the -cb value, just set auditlog to a file for dafileserver? we determined with that spreadsheet that we should have enough space to run for a few days without issue | 03:38 |
| auristor | great | 03:38 |
| *** psachin has joined #openstack-infra | 03:38 | |
| auristor | -auditlog <log_path> -audit-interface file | 03:42 |
| ianw | ok i've done afs01.dfw | 03:46 |
| *** rfarr has quit IRC | 03:47 | |
| *** rfarr has joined #openstack-infra | 03:47 | |
| auristor | is there logging to indicate when the vos release process begins and ends? | 03:49 |
| ianw | auristor: yep, it varies depending on the type of mirroring (rsyncing, using reprepro, etc) but all the cron jobs log their output. unfortunately not publicly logged ATM (i'm working on that ... no reason they can't be, just not done ATM) | 03:52 |
| auristor | thats ok. as long as you have access to it | 03:52 |
| ianw | ok, afs01/02.dfw & afs01.ord all now have audit logging. i will just update our etherpad with the details & how to turn it off, in case i get hit by a bus :) | 03:54 |
| *** ramishra has joined #openstack-infra | 03:57 | |
| ianw | #status log AFS auditlog enabled for afs01/02.dfw and afs01.ord AFS servers. logging to /opt/dafileserver.audit.log. notes, including details on how to disable again @ https://etherpad.openstack.org/p/opendev-mirror-afs | 03:58 |
| openstackstatus | ianw: finished logging | 03:58 |
| *** bhavikdbavishi1 has joined #openstack-infra | 04:02 | |
| *** bhavikdbavishi has quit IRC | 04:04 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:04 | |
| *** yamamoto has quit IRC | 04:08 | |
| *** yamamoto has joined #openstack-infra | 04:10 | |
| *** kjackal has joined #openstack-infra | 04:11 | |
| *** virendra-sharma has joined #openstack-infra | 04:12 | |
| openstackgerrit | Merged opendev/system-config master: letsencrypt-install-txt-record: skip disabled hosts https://review.opendev.org/668344 | 04:24 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Experimental kafs support https://review.opendev.org/623974 | 04:25 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ubuntu-kernel: role to use Ubuntu mainline kernels https://review.opendev.org/665057 | 04:25 |
| AJaeger | config-core, please review https://review.opendev.org/#/c/667900/ and https://review.opendev.org/667437 and https://review.opendev.org/668045 | 04:33 |
| *** ykarel has joined #openstack-infra | 04:42 | |
| *** yamamoto has quit IRC | 04:43 | |
| openstackgerrit | Merged openstack/project-config master: Fix ACLs for retired ansible-role-tripleo-cookiecutter https://review.opendev.org/667900 | 04:50 |
| openstackgerrit | Merged openstack/project-config master: Removed unused ACL https://review.opendev.org/667437 | 04:50 |
| *** ricolin_ has joined #openstack-infra | 05:03 | |
| *** ricolin has quit IRC | 05:06 | |
| *** yamamoto has joined #openstack-infra | 05:09 | |
| *** ykarel_ has joined #openstack-infra | 05:09 | |
| *** ykarel has quit IRC | 05:12 | |
| *** bradm has quit IRC | 05:13 | |
| *** udesale has joined #openstack-infra | 05:18 | |
| *** ykarel_ is now known as ykarel|meeting | 05:20 | |
| *** sshnaidm_ is now known as sshnaidm|ruck | 05:27 | |
| *** kjackal has quit IRC | 05:27 | |
| *** ociuhandu has joined #openstack-infra | 05:40 | |
| *** ykarel|meeting is now known as ykarel | 05:41 | |
| *** jbadiapa has joined #openstack-infra | 05:45 | |
| *** rpittau|mtg is now known as rpittau|afk | 05:49 | |
| *** virendra-sharma has quit IRC | 05:49 | |
| *** ociuhandu has quit IRC | 05:53 | |
| *** yamamoto has quit IRC | 05:53 | |
| *** ociuhandu has joined #openstack-infra | 05:54 | |
| *** ociuhandu has quit IRC | 05:57 | |
| *** ociuhandu has joined #openstack-infra | 05:58 | |
| *** udesale has quit IRC | 05:59 | |
| *** kjackal has joined #openstack-infra | 06:04 | |
| *** snapiri has joined #openstack-infra | 06:08 | |
| *** ricolin_ has quit IRC | 06:09 | |
| *** ricolin has joined #openstack-infra | 06:09 | |
| AJaeger | thanks, ianw ! | 06:10 |
| *** AJaeger has quit IRC | 06:11 | |
| *** AJaeger has joined #openstack-infra | 06:13 | |
| *** dpawlik has joined #openstack-infra | 06:15 | |
| *** pgaxatte has joined #openstack-infra | 06:27 | |
| *** yamamoto has joined #openstack-infra | 06:31 | |
| *** pgaxatte has quit IRC | 06:31 | |
| *** pgaxatte has joined #openstack-infra | 06:32 | |
| *** ociuhandu has quit IRC | 06:36 | |
| *** yamamoto has quit IRC | 06:41 | |
| *** slaweq has joined #openstack-infra | 06:48 | |
| *** ccamacho has joined #openstack-infra | 06:50 | |
| *** ccamacho has quit IRC | 06:50 | |
| *** ccamacho has joined #openstack-infra | 06:50 | |
| *** rcernin has quit IRC | 06:55 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch ovh/GRA1 to opendev.org mirror https://review.opendev.org/668361 | 06:57 |
| ianw | fungi / anyone: ^ a short sync update; nothing that is not also noted in https://etherpad.openstack.org/p/opendev-mirror-afs | 06:58 |
| ianw | 1) note afs servers have audit logging right now. might be easier to work with auristor in daylight hours on any required extracts | 06:59 |
| ianw | 2) mirror.gra1.ovh.opendev.org should be operational and configured basically exactly the same as iad.rax. It is logging to 6667 (iad.rax 6666) on our netconsole capture host | 06:59 |
| ianw | it is running upstream 5.2-rc7 kernel, which should have the afs fixes in it. iad.rax still running our hand-built afs-next kernel | 07:00 |
| *** ginopc has joined #openstack-infra | 07:01 | |
| ianw | clarkb: i'd say for new hosts setup as opendev.org mirror using bionic but with openafs (1.8.3 from our ppa). that is all in system-config -- so basically just copy out mirror.rax.dfw.opendev.org | 07:02 |
| *** xek has joined #openstack-infra | 07:05 | |
| AJaeger | ianw: btw. what about https://review.opendev.org/663854 - do we want that? Then it needs a rebase... | 07:08 |
| *** tesseract has joined #openstack-infra | 07:10 | |
| ianw | AJaeger: ah, yeah, we could switch in rax.ord -- that's openafs 1.8.3 based like dfw.rax | 07:11 |
| *** iurygregory has joined #openstack-infra | 07:15 | |
| AJaeger | want to base it on top of 668361? Your decision on what's the best way forward... | 07:19 |
| *** yamamoto has joined #openstack-infra | 07:19 | |
| *** lucasagomes has joined #openstack-infra | 07:19 | |
| *** tosky has joined #openstack-infra | 07:25 | |
| ianw | trying to remember now where i got to with ord server | 07:26 |
| openstackgerrit | Maksim Malchuk proposed openstack/diskimage-builder master: Cleanup: remove useless statement https://review.opendev.org/668372 | 07:30 |
| *** witek has joined #openstack-infra | 07:40 | |
| *** pcaruana has joined #openstack-infra | 07:43 | |
| *** rpittau|afk is now known as rpittau | 07:44 | |
| *** ykarel is now known as ykarel|lunch | 07:46 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch RAX ORD mirrors to new opendev.org mirrors https://review.opendev.org/663854 | 07:50 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch ovh/GRA1 to opendev.org mirror https://review.opendev.org/668361 | 07:50 |
| ianw | AJeager / clarkb: ^ I think that's the way forward ... dfw has been running with openafs 1.8.3 without issue AFAIK. so we can switch in ORD too with that; just leave our kafs testing on the two hosts (one local-ish, one remote-ish) | 07:51 |
| *** ralonsoh has joined #openstack-infra | 07:52 | |
| AJaeger | ianw: Thanks, LGTM for that change ;) | 07:59 |
| *** jpich has joined #openstack-infra | 07:59 | |
| openstackgerrit | Natal Ngétal proposed openstack/diskimage-builder master: [Configuration] Switch to stestr. https://review.opendev.org/629414 | 08:12 |
| *** ykarel|lunch is now known as ykarel | 08:14 | |
| *** pkopec has joined #openstack-infra | 08:15 | |
| *** Emine has joined #openstack-infra | 08:20 | |
| *** tkajinam has quit IRC | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:40 | |
| *** ykarel is now known as ykarel|meeting | 08:43 | |
| *** ricolin_ has joined #openstack-infra | 08:45 | |
| *** ricolin_ has quit IRC | 08:47 | |
| *** ricolin_ has joined #openstack-infra | 08:47 | |
| *** ricolin has quit IRC | 08:48 | |
| *** rfarr has quit IRC | 08:55 | |
| *** udesale has joined #openstack-infra | 08:55 | |
| *** rfarr has joined #openstack-infra | 08:55 | |
| *** apetrich has joined #openstack-infra | 08:59 | |
| *** ykarel has joined #openstack-infra | 09:05 | |
| *** ykarel|meeting has quit IRC | 09:05 | |
| *** ricolin_ is now known as ricolin | 09:07 | |
| *** ykarel_ has joined #openstack-infra | 09:09 | |
| *** ykarel_ is now known as ykarel|meeting | 09:10 | |
| *** ykarel has quit IRC | 09:10 | |
| *** ykarel|meeting is now known as ykarel | 09:11 | |
| *** ricolin has quit IRC | 09:20 | |
| *** bhavikdbavishi has quit IRC | 09:27 | |
| *** yamamoto has quit IRC | 09:31 | |
| *** Lucas_Gray has joined #openstack-infra | 09:35 | |
| *** ykarel has quit IRC | 09:40 | |
| *** yamamoto has joined #openstack-infra | 09:40 | |
| *** adrianreza has quit IRC | 09:48 | |
| *** whoami-rajat has joined #openstack-infra | 09:52 | |
| *** ociuhandu has joined #openstack-infra | 09:55 | |
| *** roman_g has joined #openstack-infra | 09:57 | |
| *** ociuhandu has quit IRC | 10:00 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** kjackal has quit IRC | 10:08 | |
| *** Wryhder has joined #openstack-infra | 10:15 | |
| *** Lucas_Gray has quit IRC | 10:16 | |
| *** Wryhder is now known as Lucas_Gray | 10:16 | |
| *** jaosorior has joined #openstack-infra | 10:17 | |
| *** yamamoto has quit IRC | 10:21 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:36 | |
| *** bhavikdbavishi has quit IRC | 10:38 | |
| *** kjackal has joined #openstack-infra | 10:44 | |
| *** lucasagomes has quit IRC | 10:49 | |
| *** yamamoto has joined #openstack-infra | 10:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:57 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** lucasagomes has joined #openstack-infra | 11:01 | |
| *** Lucas_Gray has quit IRC | 11:02 | |
| *** Lucas_Gray has joined #openstack-infra | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| *** yamamoto has quit IRC | 11:11 | |
| openstackgerrit | inspurericzhang proposed openstack/diskimage-builder master: Mailing lists change openstack-dev to openstack-discuss https://review.opendev.org/668435 | 11:17 |
| openstackgerrit | Marios Andreou proposed openstack/diskimage-builder master: WIP: Adds release->rhel8 for ironic-agent pkg-map https://review.opendev.org/668436 | 11:18 |
| *** rascasoft has quit IRC | 11:30 | |
| *** rascasoft has joined #openstack-infra | 11:33 | |
| *** pcaruana has quit IRC | 11:36 | |
| *** bhavikdbavishi has quit IRC | 11:37 | |
| openstackgerrit | inspurericzhang proposed openstack/reviewstats master: Mailing lists change openstack-dev to openstack-discuss https://review.opendev.org/668446 | 11:38 |
| *** bhavikdbavishi has joined #openstack-infra | 11:44 | |
| *** yamamoto has joined #openstack-infra | 11:47 | |
| *** ykarel has joined #openstack-infra | 11:49 | |
| *** ykarel is now known as ykarel|meeting | 11:49 | |
| *** rh-jelabarre has joined #openstack-infra | 11:50 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Allow proceeding with requests on quota exceeded https://review.opendev.org/667371 | 11:51 |
| *** jaosorior has quit IRC | 11:56 | |
| *** jaosorior has joined #openstack-infra | 11:56 | |
| *** jaosorior has quit IRC | 11:57 | |
| *** jaosorior has joined #openstack-infra | 11:57 | |
| *** dchen has quit IRC | 11:59 | |
| mordred | morning AJaeger - getting back up to speed, anything you want me to look at while I'm waking up? | 12:03 |
| sshnaidm|ruck | I have an error with docker proxy I suppose: HTTPError: 401 Client Error: Unauthorized for url: http://mirror.bhs1.ovh.openstack.org:8082/v2/tripleomaster/centos-binary-mistral-api/blobs/sha256:469cc40974973b41dc9d82f4cfe406a6c630d5952f86f819cd853d3785d4fc5f | 12:14 |
| sshnaidm|ruck | could it be proxy failure or something else..? | 12:15 |
| mordred | sshnaidm|ruck: yeah - I'm still coffeeing, so I might be wrong, but what that usually means is that something has caused docker to talk to dockerhub instead of the proxy (the proxy hub takes credentials, and docker thinks the proxy is dockerhub, so if it falls back to _actual_ dockerhub it uses the credentials to do so which are not real credentials) | 12:18 |
| *** rosmaita has joined #openstack-infra | 12:19 | |
| openstackgerrit | Marios Andreou proposed openstack/diskimage-builder master: WIP: Adds release->rhel8 for ironic-agent pkg-map https://review.opendev.org/668436 | 12:21 |
| *** Lucas_Gray has quit IRC | 12:23 | |
| *** Lucas_Gray has joined #openstack-infra | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:28 | |
| *** electrofelix has joined #openstack-infra | 12:30 | |
| *** piotrowskim has joined #openstack-infra | 12:31 | |
| *** tdasilva has joined #openstack-infra | 12:36 | |
| *** eharney has quit IRC | 12:38 | |
| *** Wryhder has joined #openstack-infra | 12:41 | |
| *** Lucas_Gray has quit IRC | 12:43 | |
| *** Wryhder is now known as Lucas_Gray | 12:43 | |
| *** ykarel|meeting is now known as ykarel | 12:50 | |
| *** rfarr_ has joined #openstack-infra | 12:53 | |
| sshnaidm|ruck | is there a short way to retrieve all gate job names for a specific project? maybe via zuul api | 12:54 |
| *** ekultails has joined #openstack-infra | 12:55 | |
| *** aaronsheffield has joined #openstack-infra | 12:55 | |
| *** rfarr has quit IRC | 12:56 | |
| *** yamamoto has quit IRC | 12:57 | |
| *** jaosorior has quit IRC | 13:03 | |
| *** jaosorior has joined #openstack-infra | 13:03 | |
| *** auristor has quit IRC | 13:05 | |
| *** Lucas_Gray has quit IRC | 13:06 | |
| *** Lucas_Gray has joined #openstack-infra | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:10 | |
| *** yamamoto has quit IRC | 13:14 | |
| *** eharney has joined #openstack-infra | 13:15 | |
| *** sthussey has joined #openstack-infra | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:27 | |
| *** lseki has joined #openstack-infra | 13:29 | |
| *** iurygregory has quit IRC | 13:29 | |
| AJaeger | morning mordred , the stack at https://review.opendev.org/#/q/status:open+++topic:forge-publish is something I'd like to see in ;) | 13:29 |
| AJaeger | config-core, please review also https://review.opendev.org/#/c/665012/ and https://review.opendev.org/667119 | 13:31 |
| mordred | AJaeger: that whole stack looks nice | 13:34 |
| AJaeger | thanks, mordred - tobias-urdin did the work, I just split up ;) | 13:34 |
| *** kjackal has quit IRC | 13:34 | |
| mordred | AJaeger, tobias-urdin: well, it was very easy to follow, so thanks! | 13:35 |
| AJaeger | mordred, corvus , I'd also like a review for two changes to switch to opendev, see https://review.opendev.org/667820 and https://review.opendev.org/667813 , please? | 13:38 |
| *** bhavikdbavishi has quit IRC | 13:39 | |
| tobias-urdin | thanks for reviews and to AJaeger for completing it :) | 13:40 |
| AJaeger | tobias-urdin, mordred , due to the depends-on, we need to merge them one-by-one - I'll add my own +A to move them forward | 13:42 |
| mordred | AJaeger: cool | 13:42 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/openstack-zuul-jobs master: Add binary dependencies for building TeX docs https://review.opendev.org/668464 | 13:42 |
| AJaeger | thanks, mordred. | 13:43 |
| *** kjackal has joined #openstack-infra | 13:44 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Add openstack-tox-functional-py37 job https://review.opendev.org/667119 | 13:44 |
| AJaeger | mordred: you might also be interested in fungi's changes for ansible-lint, see https://review.opendev.org/#/q/status:open+project:zuul/zuul-jobs++topic:ansible-lint . Feel free to -1 those you disagree with | 13:44 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Set python_version: 3.7 for openstack-tox-functional-py37 https://review.opendev.org/668466 | 13:45 |
| openstackgerrit | Merged openstack/project-config master: Remove nodepool builder config for control plane images https://review.opendev.org/665012 | 13:47 |
| *** yamamoto has joined #openstack-infra | 13:50 | |
| sshnaidm|ruck | I'd like to add configure-mirrors role to tox-molecule job to use docker mirros | 13:52 |
| sshnaidm|ruck | should I create a different parent for this job that uses this role or what is better to do? | 13:53 |
| sshnaidm|ruck | zbr, clarkb ^^ | 13:53 |
| openstackgerrit | Merged zuul/zuul-jobs master: Prepare: Use PDK to build puppet module https://review.opendev.org/667590 | 13:53 |
| AJaeger | sshnaidm|ruck: Isn't configure-mirror used by all tox jobs? | 13:54 |
| *** jcoufal has joined #openstack-infra | 13:54 | |
| zbr | sshnaidm|ruck: enabling support for mirrors is likely to break it? if not altering the tox-molecule job may be best approach. | 13:54 |
| sshnaidm|ruck | AJaeger, how can I check it? I'm afraid I don't use mirror in molecule job | 13:54 |
| mordred | AJaeger: I have left votes and comments | 13:55 |
| zbr | btw, I have a more generic question: why not using a generic http-proxy for caching? molecule can make use of http(s)_proxy by default. | 13:56 |
| AJaeger | sshnaidm|ruck: check the log file ;) | 13:56 |
| AJaeger | sshnaidm|ruck: if you cannot find it, please point me to a review | 13:57 |
| AJaeger | mordred: great | 13:57 |
| *** yamamoto has quit IRC | 13:57 | |
| sshnaidm|ruck | AJaeger, for example this job: http://logs.openstack.org/66/668166/3/check/tripleo-ansible-centos-7-molecule-tripleo-bootstrap/5672866/ how can I figure out that I used docker proxy (or not) | 14:00 |
| sshnaidm|ruck | AJaeger, I see configure-mirrors is running, but how to be sure it's active for docker | 14:02 |
| AJaeger | sshnaidm|ruck: can't help with the "usage" part. Is there a way to query docker for a proxy? | 14:04 |
| *** dpawlik has quit IRC | 14:05 | |
| zbr | sshnaidm|ruck: is not, AFAIK docker needs individual config to use http proxy, it does *not* make use of http_proxy env vars (99.9% about that) | 14:05 |
| zbr | needs to be added manually in its json config | 14:05 |
| zbr | but this feature should be part of the role that configures docker, not needed to be separated from it IMHO | 14:05 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Install PDK dependencies for puppet building https://review.opendev.org/667589 | 14:06 |
| zbr | i have a ? reg use of requirement-check which apparently is called only when some req*.txt files are touched. and apparently you discover that they are broken only when you have to modify one of the files. faced this recently. any good solution for avoiding that? enabling requirements-check on any change seems like a waste of resources to me... | 14:06 |
| *** pcaruana has joined #openstack-infra | 14:06 | |
| openstackgerrit | Merged openstack/project-config master: Install PDK dependencies for puppet building https://review.opendev.org/667595 | 14:07 |
| AJaeger | corvus: thanks for reviews, I'll send out an announcement about https://review.opendev.org/667813 | 14:07 |
| openstackgerrit | Merged zuul/zuul-website master: Change git to opendev.org URLs https://review.opendev.org/667820 | 14:08 |
| clarkb | zbr we dont usr a generic http proxy because we cant filter source addresses on the proxy and would have to be a wideopen proxy on the internet which we dont want | 14:09 |
| clarkb | instead we reverse proxy things that likely wont have problems being proxied | 14:10 |
| *** pgaxatte has quit IRC | 14:10 | |
| clarkb | (we cant filter source addrs because we dont know what the valid ip range for each cloud region is) | 14:10 |
| zbr | clarkb: clouds made that much harder. maybe enabling auth could be a solution? somehow I find the need to re-configure and maintain more and more mirrors as an activity that does not scale very well. | 14:12 |
| *** pgaxatte has joined #openstack-infra | 14:12 | |
| corvus | there is no auth that we could keep secret | 14:12 |
| AJaeger | corvus: email send to zuul-announce... | 14:14 |
| corvus | (conceivably, we could perhaps put a proxy on a private neutron network that we attach every instance to, but i'm not sure whether we would be able to do that in every cloud region that we use) | 14:15 |
| clarkb | ya I think rax and ovh wouldnt be able to do that | 14:19 |
| clarkb | (but may be wrong) | 14:19 |
| *** kaiokmo has quit IRC | 14:19 | |
| *** iurygregory has joined #openstack-infra | 14:19 | |
| *** jamesmcarthur has joined #openstack-infra | 14:20 | |
| *** hemna has quit IRC | 14:22 | |
| AJaeger | clarkb: are you around and want to watch ianw's new opendev mirrors, see https://review.opendev.org/663854 for change, please | 14:22 |
| clarkb | AJaeger: thanks, I do intend on catching up on that today as I'll need to deploy one as soon aa donnyd reports fortnebula is ready again | 14:26 |
| corvus | AJaeger: approved | 14:27 |
| corvus | (the email) | 14:27 |
| AJaeger | strange, http://zuul.openstack.org/stream/a52ccb5ff6124ea0a01fe62995330707?logfile=console.log gives me since several minutes only "End of stream", repeated every minute? Are those repeated node start failures for change https://review.opendev.org/627534 ? | 14:27 |
| AJaeger | thanks, corvus | 14:28 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use PDK to build puppet module https://review.opendev.org/627534 | 14:28 |
| AJaeger | ah, now it's finished.... | 14:28 |
| AJaeger | strange - log file exists, just streaming problem? | 14:28 |
| fungi | AJaeger: if it's a job which reboots the node or causes the console stream process to die for other reasons, that's a possible explanation | 14:29 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 14:29 |
| AJaeger | fungi: I got log files in the end... | 14:30 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 14:30 |
| fungi | well, logs still get collected | 14:30 |
| fungi | just the console stream won't produce any output and you'll see "end of stream" there instead while the job runs | 14:31 |
| fungi | infra-root: reminder, i'm out all week and not around the computer much | 14:31 |
| mordred | fungi: have fun! I enjoyed doing that last week | 14:31 |
| AJaeger | fungi: have fun! thanks for explanation | 14:31 |
| donnyd | clarkb I am hopeful it will be up in the next hour or so | 14:33 |
| donnyd | Also can I ask you to switch back from raw to normal images | 14:33 |
| donnyd | I got the storage servers to run 2x faster, just needed to use a different backend... Hence the reason for rebuild... | 14:34 |
| *** udesale has quit IRC | 14:34 | |
| *** udesale has joined #openstack-infra | 14:34 | |
| *** dklyle has joined #openstack-infra | 14:37 | |
| clarkb | donnyd: sure, qcow2 images? | 14:37 |
| clarkb | I've got some meetings today and the kids have a dentist visit so I'll be juggling that too, but will do my best to get things back up and into a testable state | 14:38 |
| clarkb | fungi: enjoy the vacation time | 14:38 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove bindep role for Puppet build job https://review.opendev.org/627554 | 14:40 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Test install-pdk-dependencies role https://review.opendev.org/668337 | 14:40 |
| openstackgerrit | Merged openstack/project-config master: Remove bindep role for Puppet build job https://review.opendev.org/627566 | 14:42 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add testing job to zuul-jobs https://review.opendev.org/668483 | 14:46 |
| AJaeger | corvus: here's the job addition ^ | 14:46 |
| *** ginopc has quit IRC | 14:46 | |
| AJaeger | thanks for reviews! | 14:47 |
| *** rajinir has joined #openstack-infra | 14:55 | |
| *** pcaruana has quit IRC | 14:56 | |
| *** cmorpheus is now known as cmurphy | 14:56 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 14:58 |
| AJaeger | mordred: want to review https://review.opendev.org/668337 and https://review.opendev.org/668483 as well, please? A small job to test one of the two just added roles | 14:59 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 15:01 |
| donnyd | clarkb: yea, qcows will probably be better for internety things anyways | 15:01 |
| *** ykarel is now known as ykarel|away | 15:02 | |
| donnyd | clarkb: no worries. Thank you for continuing to be helpful in getting this thing running well | 15:02 |
| *** auristor has joined #openstack-infra | 15:05 | |
| *** josephrsandoval has joined #openstack-infra | 15:06 | |
| clarkb | donnyd: no problem and really thank you for helping us too. Just let me know when I should reupload images and run the nova unittests again | 15:08 |
| *** diablo_rojo has joined #openstack-infra | 15:09 | |
| *** factor has joined #openstack-infra | 15:09 | |
| *** icarusfactor has quit IRC | 15:10 | |
| *** icarusfactor has joined #openstack-infra | 15:11 | |
| *** factor has quit IRC | 15:11 | |
| donnyd | clarkb: Is there a place I could download the image and then run a script or cloud-config? | 15:12 |
| clarkb | donnyd: there is! one sec | 15:12 |
| donnyd | Nice!!! | 15:12 |
| clarkb | donnyd: https://nb01.openstack.org/images/ and https://nb02.openstack.org/images/ have our imagse you can download them and confirm the hashes, then reupload and boot them. Note you need to set config drive to true and it will configure ssh keys for the root user | 15:13 |
| clarkb | I think https may not have a properly signed cert on those hosts. ianw ^ we should totally LE those now that lE is a thing we can do | 15:13 |
| *** pgaxatte has quit IRC | 15:14 | |
| *** factor has joined #openstack-infra | 15:14 | |
| *** icarusfactor has quit IRC | 15:15 | |
| donnyd | And what nova unittests do you run? Just want to make sure I am running the metrics against the same thing you were doing last week | 15:17 |
| clarkb | donnyd: last week I ran the full nova unittest suite with tox. The basic process is git clone nova, run bindep `/usr/bindep-env/bin/bindep -f /path/to/nova/bindep.txt test`, take that output and apt-get install the results, run nova/tools/test-setup.sh, then run tox -epy27 in nova dir | 15:18 |
| *** ricolin has joined #openstack-infra | 15:20 | |
| *** ijw has joined #openstack-infra | 15:20 | |
| donnyd | Ok I will give it a spin | 15:20 |
| clarkb | donnyd: all of this was ont he ubuntu-bionic image which is where we run those tests with zuul | 15:21 |
| *** icarusfactor has joined #openstack-infra | 15:21 | |
| *** factor has quit IRC | 15:24 | |
| *** lucasagomes has quit IRC | 15:31 | |
| *** ekultails has quit IRC | 15:31 | |
| *** ykarel|away has quit IRC | 15:35 | |
| *** goldyfruit has quit IRC | 15:39 | |
| *** jaosorior has quit IRC | 15:42 | |
| *** ijw_ has joined #openstack-infra | 15:47 | |
| *** ekultails has joined #openstack-infra | 15:48 | |
| *** ijw has quit IRC | 15:51 | |
| *** weshay is now known as weshay_PTO | 15:52 | |
| *** pcaruana has joined #openstack-infra | 15:53 | |
| *** tdasilva has quit IRC | 15:56 | |
| *** e0ne has quit IRC | 16:00 | |
| *** josephrsandoval has quit IRC | 16:01 | |
| *** whoami-rajat has quit IRC | 16:01 | |
| *** ramishra has quit IRC | 16:02 | |
| *** jpich has quit IRC | 16:10 | |
| *** ricolin has quit IRC | 16:13 | |
| *** ykarel|away has joined #openstack-infra | 16:13 | |
| *** ykarel|away has quit IRC | 16:18 | |
| *** Lucas_Gray has quit IRC | 16:19 | |
| *** mattw4 has joined #openstack-infra | 16:20 | |
| *** ijw_ has quit IRC | 16:21 | |
| *** goldyfruit has joined #openstack-infra | 16:21 | |
| *** eernst has joined #openstack-infra | 16:21 | |
| *** eernst has quit IRC | 16:23 | |
| *** eernst has joined #openstack-infra | 16:23 | |
| *** mattw4 has quit IRC | 16:25 | |
| *** ricolin has joined #openstack-infra | 16:25 | |
| *** eernst has quit IRC | 16:26 | |
| *** eernst has joined #openstack-infra | 16:29 | |
| *** rlandy is now known as rlandy|brb | 16:30 | |
| *** chandankumar is now known as raukadah | 16:34 | |
| *** rpittau is now known as rpittau|afk | 16:36 | |
| *** Emine has quit IRC | 16:39 | |
| *** mattw4 has joined #openstack-infra | 16:43 | |
| *** ardaguclu has joined #openstack-infra | 16:46 | |
| *** witek has quit IRC | 16:46 | |
| zbr | cfp.openstack.org persistently gives "There was a problem with our server, please contact admin." | 16:47 |
| zbr | after which it redirects me to a callback page that appears empty | 16:48 |
| *** udesale has quit IRC | 16:48 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Fix docker install on fedora https://review.opendev.org/668527 | 16:52 |
| *** ricolin has quit IRC | 16:53 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| *** ijw has joined #openstack-infra | 17:06 | |
| clarkb | zbr: you can file bugs for that at https://bugs.launchpad.net/openstack-org I'll also let them know | 17:09 |
| *** mattw4 has quit IRC | 17:10 | |
| zbr | clarkb: thanks for the hint. is there a specific channel for the subject? For example I was wondering when is someone added to /community/speakers -- marking yourself available does not list you there. | 17:11 |
| clarkb | zbr: it is run by foundation staff and not the infra team. I don't think they have achannel but are typically responsive to email and bug reports | 17:12 |
| zbr | thanks, i will use email in this case. | 17:12 |
| jamesmcarthur | zbr: there is no specific channel for speakers, but you can send bug requests/reports to speakersupport@openstack.org | 17:12 |
| *** ociuhandu has quit IRC | 17:13 | |
| *** mattw4 has joined #openstack-infra | 17:13 | |
| zbr | i will send may questions there | 17:13 |
| jamesmcarthur | zbr: regarding the call back error, we had an issue that has subsequently been fixed. | 17:13 |
| jamesmcarthur | It's possible you have a cached version. Could you try clearing your cache? | 17:13 |
| *** rh-jelabarre has quit IRC | 17:13 | |
| clarkb | donnyd: just a note that I'm not able to hit keystone currently | 17:13 |
| clarkb | (I'm finally back to computer and was goign to upload a qcow2, let me know if I can help) | 17:14 |
| *** rh-jelabarre has joined #openstack-infra | 17:15 | |
| donnyd | yea... still going through gyrations over here | 17:15 |
| donnyd | LOL | 17:15 |
| clarkb | donnyd: no worries. Ping me when you think it is ready for me to have a go at it | 17:15 |
| donnyd | https://bugs.launchpad.net/tripleo/+bug/1834857 | 17:15 |
| openstack | Launchpad bug 1834857 in tripleo "Podman fails to start Glance API container mounted with NFS backend" [Undecided,New] | 17:15 |
| *** ralonsoh has quit IRC | 17:16 | |
| donnyd | I am working around it now... Fingers crossed it will on be a couple more hours | 17:16 |
| donnyd | I surely will | 17:16 |
| donnyd | I was able to get the images pulled, so they will be uploaded stat once its back up | 17:16 |
| clarkb | donnyd: note that those images are refreshed by our tooling daily so no need toupload them as "permanent" images | 17:16 |
| *** smarcet has joined #openstack-infra | 17:17 | |
| clarkb | (and I"m happy to upload them myself for testing) | 17:17 |
| clarkb | mordred: the other item on my todo list is cleaning up the control plane cloud images from nodepool | 17:18 |
| clarkb | mordred: I didn't want to start really digging into that until you were around | 17:19 |
| mordred | clarkb: cool- I'm about to lunch, but will be around to help once I'm no longer lunching | 17:19 |
| clarkb | mordred: https://review.opendev.org/#/c/665018/ and its depends on are the things to look at I think | 17:19 |
| clarkb | enjoy lunch! | 17:19 |
| * mordred is gonna lunch all the lunches | 17:22 | |
| *** bobh has joined #openstack-infra | 17:24 | |
| *** ardaguclu has quit IRC | 17:27 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix docker install on fedora https://review.opendev.org/668527 | 17:29 |
| openstackgerrit | Merged openstack/project-config master: Switch RAX ORD mirrors to new opendev.org mirrors https://review.opendev.org/663854 | 17:29 |
| clarkb | AJaeger: ^ fyi. i'll keep an eye on it | 17:30 |
| *** efried has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack/project-config master: Switch ovh/GRA1 to opendev.org mirror https://review.opendev.org/668361 | 17:30 |
| *** ardaguclu has joined #openstack-infra | 17:32 | |
| AJaeger | thanks, clarkb | 17:33 |
| *** rfarr__ has joined #openstack-infra | 17:33 | |
| *** ardaguclu has quit IRC | 17:34 | |
| *** ardaguclu has joined #openstack-infra | 17:34 | |
| AJaeger | config-core, could review two changes to finish the puppet build changes, please? https://review.opendev.org/668337 and https://review.opendev.org/668483 as well, please? These add a small job for testing one of the roles. | 17:35 |
| *** rfarr_ has quit IRC | 17:36 | |
| *** rlandy|brb is now known as rlandy | 17:36 | |
| clarkb | AJaeger: first is approved. I'll approve the other once the first merges (due to the depends on) | 17:37 |
| AJaeger | thanks, clarkb | 17:37 |
| *** electrofelix has quit IRC | 17:38 | |
| *** rfarr_ has joined #openstack-infra | 17:40 | |
| *** rfarr__ has quit IRC | 17:43 | |
| *** xek has quit IRC | 17:45 | |
| *** jamesmcarthur has quit IRC | 17:47 | |
| *** ociuhandu has joined #openstack-infra | 17:53 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Test install-pdk-dependencies role https://review.opendev.org/668337 | 17:54 |
| *** psachin has quit IRC | 17:55 | |
| *** harlowja has joined #openstack-infra | 17:58 | |
| *** ociuhandu has quit IRC | 18:02 | |
| prometheanfire | is there a reason the base debian element in dib installs cloud-init, cloud-initramfs-growroot, etc? | 18:03 |
| clarkb | prometheanfire: I'm guessing because they are needed for useful cloud images | 18:05 |
| clarkb | note the minimal element does not | 18:05 |
| clarkb | also the non minimal elements tend to pull the upstream images and update them | 18:05 |
| clarkb | so could just be from that | 18:05 |
| prometheanfire | minimal element is also not systemd | 18:07 |
| *** bobh has quit IRC | 18:07 | |
| prometheanfire | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/debian-minimal/root.d/75-debian-minimal-baseinstall#L69 | 18:08 |
| prometheanfire | or do docs need updating... | 18:08 |
| clarkb | prometheanfire: it is on our VMs | 18:08 |
| clarkb | I don't think there is any avoiding it on debian | 18:08 |
| prometheanfire | ya, looks like docs were not updated :| | 18:08 |
| prometheanfire | Note that this element installs systemd-sysv as the init system | 18:09 |
| prometheanfire | https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/debian-minimal | 18:09 |
| prometheanfire | for that, meh | 18:09 |
| *** igordc has joined #openstack-infra | 18:09 | |
| openstackgerrit | Merged zuul/zuul master: Add extra-config-paths tenant config option https://review.opendev.org/668214 | 18:09 |
| clarkb | thats still systemd and runs units right? I don't know what all the options are for debian but my undersatnding is modern debian is systemd required | 18:10 |
| clarkb | but ya usually the non minimal elements take an upstream image and modify it (so you get all their opinions) and the minimal element attempts to be minimal | 18:10 |
| prometheanfire | yep, it's just docs being out of date | 18:11 |
| openstackgerrit | Merged openstack/project-config master: Add testing job to zuul-jobs https://review.opendev.org/668483 | 18:14 |
| *** igordc has quit IRC | 18:14 | |
| *** igordc has joined #openstack-infra | 18:15 | |
| *** igordc has quit IRC | 18:17 | |
| *** jamesmcarthur has joined #openstack-infra | 18:24 | |
| *** diablo_rojo has quit IRC | 18:30 | |
| *** weifan has joined #openstack-infra | 18:32 | |
| *** eernst has quit IRC | 18:34 | |
| *** diablo_rojo has joined #openstack-infra | 18:37 | |
| *** eernst has joined #openstack-infra | 18:37 | |
| *** kaiokmo has joined #openstack-infra | 18:38 | |
| openstackgerrit | Graham Hayes proposed zuul/nodepool master: Implement an Azure driver https://review.opendev.org/554432 | 18:45 |
| *** weifan has quit IRC | 18:49 | |
| clarkb | mordred: https://review.opendev.org/#/c/667986/ is another one you might make a quick glance at. In rebuilding gitea06 I ended up in a corrupt database table sitaution and I think that change will avoid that in the future by only backing up (then restoring) the gitea db | 18:50 |
| clarkb | mordred: if there is a reason to keep backing up the other dbs in mysql let me know (otherwise I'll likely approve that change) | 18:50 |
| *** ijw has quit IRC | 18:56 | |
| *** ijw has joined #openstack-infra | 18:56 | |
| *** jtomasek has joined #openstack-infra | 18:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Reload config on extra config files changes https://review.opendev.org/668548 | 18:58 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 18:59 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: set default sources conf for buster as it now has a release https://review.opendev.org/668549 | 19:00 |
| *** ardaguclu has left #openstack-infra | 19:08 | |
| *** ociuhandu has joined #openstack-infra | 19:12 | |
| *** ijw_ has joined #openstack-infra | 19:31 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: install gnupg2 by default in debian-minimal https://review.opendev.org/668554 | 19:34 |
| *** ijw has quit IRC | 19:35 | |
| *** bobh has joined #openstack-infra | 19:39 | |
| *** weifan has joined #openstack-infra | 19:49 | |
| *** yamamoto has joined #openstack-infra | 19:50 | |
| clarkb | prometheanfire: dpkg doesn't already pull that in? huh | 19:53 |
| prometheanfire | clarkb: nope, and it's needed early enough that it needs to happen there, though it could be installed in the 02-gpg pre-install.d file | 19:55 |
| *** yamamoto has quit IRC | 19:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Reload config on extra config files changes https://review.opendev.org/668548 | 20:00 |
| *** hamzy has quit IRC | 20:02 | |
| *** dciabrin has quit IRC | 20:03 | |
| *** hamzy has joined #openstack-infra | 20:07 | |
| *** jtomasek has quit IRC | 20:08 | |
| *** rh-jelabarre has quit IRC | 20:14 | |
| *** rfarr has joined #openstack-infra | 20:14 | |
| *** eharney has quit IRC | 20:16 | |
| *** rfarr_ has quit IRC | 20:16 | |
| *** rfarr_ has joined #openstack-infra | 20:17 | |
| *** ociuhandu has quit IRC | 20:20 | |
| *** ociuhandu has joined #openstack-infra | 20:20 | |
| *** rfarr has quit IRC | 20:21 | |
| *** rh-jelabarre has joined #openstack-infra | 20:21 | |
| *** Lucas_Gray has joined #openstack-infra | 20:21 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Reload config on extra config files changes https://review.opendev.org/668548 | 20:22 |
| mordred | clarkb: reading now | 20:23 |
| mordred | clarkb: wow, they keep making mysqldump harder and harder to use properly | 20:24 |
| clarkb | mordred: fwiw I worked around it by manually editting the dump to not restore anything but the gitea db | 20:25 |
| clarkb | so I think this should be fine | 20:25 |
| clarkb | But also you know things about that stuff so figured I would aks | 20:25 |
| mordred | clarkb: +2 from me - want me to just +a it or want to babysit? | 20:26 |
| clarkb | mordred: go ahead and +a and I'll check the results tomorrow morning | 20:26 |
| clarkb | (it backs up in the middle of the night iirc) | 20:26 |
| mordred | done | 20:27 |
| clarkb | mordred: thanks@! | 20:33 |
| clarkb | and then on the control plane image cleanup. Mostly I wanted us to watch that happen as merging stuff results in it deleting images | 20:33 |
| clarkb | well watch that happen and/or double check there isn't anything we want from that | 20:34 |
| *** ociuhandu has quit IRC | 20:34 | |
| clarkb | mordred: also I discovered we want toadd at least ping6 when we get to better spot iwth that | 20:35 |
| clarkb | since launch node attempts to ping6 things after boot | 20:35 |
| mordred | clarkb: oh - yeah. ping6 seems like an important thing then :) | 20:37 |
| mordred | clarkb: I don't think there's anything we want from the images that are there, so it should be fine to delete them | 20:37 |
| mordred | but yeah - watching is good :) | 20:37 |
| *** ijw_ has quit IRC | 20:39 | |
| *** ijw has joined #openstack-infra | 20:40 | |
| *** pcaruana has quit IRC | 20:41 | |
| *** pkopec has quit IRC | 20:41 | |
| *** pkopec has joined #openstack-infra | 20:43 | |
| clarkb | ok today has been weird for me so probably won't be approving any of that today, but if people can review tomorrow should be good | 20:44 |
| *** tdasilva has joined #openstack-infra | 20:45 | |
| *** jamesmcarthur has quit IRC | 20:45 | |
| *** pkopec has quit IRC | 20:46 | |
| *** pkopec has joined #openstack-infra | 20:47 | |
| * clarkb tries to clean up some older changes that zuul is unhappy with | 20:47 | |
| *** Lucas_Gray has quit IRC | 20:50 | |
| donnyd | clarkb: finally I am getting somewhere | 20:52 |
| openstackgerrit | Merged opendev/system-config master: Only backup the gitea database on gitea hosts https://review.opendev.org/667986 | 20:53 |
| donnyd | Only took 8 hours of deployments to figure out what the issue was | 20:53 |
| clarkb | hrm ansible query if told to errors='ignore' doesn't return an empty list if it would otherwise error? but skip: true does except skip:true is deprecated? | 20:54 |
| clarkb | so how are you supposed to use query(first_found) with loop and ignore errors? | 20:54 |
| *** rfarr_ has quit IRC | 20:55 | |
| *** rfarr has joined #openstack-infra | 20:55 | |
| *** Lucas_Gray has joined #openstack-infra | 20:55 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 20:55 |
| *** jcoufal has quit IRC | 20:56 | |
| clarkb | the docs say it should return a blank list hoewver testing says that isn't the case | 20:57 |
| *** eharney has joined #openstack-infra | 20:57 | |
| mordred | clarkb: we have a cantrip for that - it's complicated - let me find you an example | 20:57 |
| clarkb | http://logs.openstack.org/10/667710/2/check/system-config-run-base/619c1e8/job-output.txt.gz#_2019-06-27_00_17_25_538312 failures for https://review.opendev.org/#/c/667710/2/playbooks/roles/unbound/tasks/main.yaml | 20:58 |
| mordred | clarkb: hrm. I can't find the example of the thing you're talking about | 20:59 |
| mordred | clarkb: I remember beating my head against that a year ago or something like that though | 21:00 |
| mordred | corvus, dmsimard: do you remember how do use loop, lookup(first_found) and to just do nothing if nothing is found? ^^ | 21:00 |
| clarkb | that contradicts https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/lookup/first_found.py#L47-L56 | 21:01 |
| clarkb | mordred: fwiw the code changes I made were based on the examples | 21:01 |
| clarkb | and I'm just updating an existing loop to use the non deprecated form of the lookup | 21:01 |
| clarkb | because our ansible run all log is full of deprecation warnings now and I find them annoying when trying to monitor ansible progress | 21:01 |
| clarkb | maybe q != query? | 21:02 |
| *** jamesmcarthur has joined #openstack-infra | 21:03 | |
| mordred | clarkb: ah - nod - I understand now. and ... maybe it's different? | 21:04 |
| *** bobh has quit IRC | 21:04 | |
| mordred | clarkb: so maybe try just using q and see if it works? | 21:04 |
| clarkb | k I'll push that up and see if it works | 21:04 |
| corvus | mordred, clarkb: is it skip=true like in https://opendev.org/opendev/system-config/src/branch/master/roles/exim/tasks/main.yaml#L14 | 21:05 |
| *** ekultails has quit IRC | 21:05 | |
| *** mattw4 has quit IRC | 21:05 | |
| clarkb | corvus: that is the deprecated form that will be removed in 2.12 | 21:05 |
| clarkb | so ya that works, its just going away in 2.12 apparently and we get tons of logging about it | 21:06 |
| *** mattw4 has joined #openstack-infra | 21:06 | |
| corvus | what? that was the *new* thing that replaced "with_first_found" | 21:06 |
| mordred | corvus: there's apparently a new new thing | 21:06 |
| *** hwoarang has quit IRC | 21:06 | |
| clarkb | corvus: the change I linked has deprecation notice details in its commit message | 21:06 |
| * corvus vocalizes thoughts in a medium not subject to permanent recording | 21:06 | |
| *** hwoarang has joined #openstack-infra | 21:09 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup deprecated ansible syntax https://review.opendev.org/667710 | 21:09 |
| *** slaweq has quit IRC | 21:11 | |
| *** tesseract has quit IRC | 21:12 | |
| *** weifan has quit IRC | 21:15 | |
| *** weifan has joined #openstack-infra | 21:17 | |
| *** weifan has quit IRC | 21:20 | |
| *** mattw4 has quit IRC | 21:25 | |
| *** mattw4 has joined #openstack-infra | 21:25 | |
| haleyb | redrobot: about a month ago i had started looking at moving the fedora-latest job to f29 from f28 and barbican was one of the consumers. any chance you can look at my test patch? https://review.opendev.org/#/c/662543/ - looks like there's some pki package changes to do, hoping you'd have a suggestion | 21:25 |
| *** dciabrin has joined #openstack-infra | 21:26 | |
| *** mattw4 has quit IRC | 21:28 | |
| clarkb | ok more data on the ansible loop thing. It does still fail using q so I think q == query. On closer inspection the same include OS specific tasks works on bridge (bionic) but not a trusty test node. The only os specific tasks are for redhat so both of them should fail the lookup then return [] | 21:29 |
| clarkb | possibly a python version problem? | 21:29 |
| clarkb | we do use python3 on bridge iirc | 21:29 |
| *** weifan has joined #openstack-infra | 21:33 | |
| *** kjackal has quit IRC | 21:39 | |
| dmsimard | mordred: apologies for the latency, on PTO | 21:42 |
| dmsimard | There's an argument to include_vars or with_first_found to skip if nothing is found iirc | 21:43 |
| dmsimard | If skip is being deprecated that's news to me | 21:44 |
| clarkb | dmsimard: it is :) | 21:44 |
| clarkb | dmsimard: [DEPRECATION WARNING]: Use errors="ignore" instead of skip. This feature will be removed in version 2.12. | 21:45 |
| clarkb | unfortunately the behavior doesn't seem to be consistent with skip | 21:45 |
| dmsimard | I suppose it's arguably similar but it's not the same yeah | 21:45 |
| *** bobh has joined #openstack-infra | 21:46 | |
| clarkb | the examples for first_found document that this should work but it does not appear to. On further investigating it may work on different platforms (possibly a python3 vs python2 behavior difference) | 21:47 |
| dmsimard | Would "ignore_errors: yes" on the task itself work ? | 21:48 |
| clarkb | only if you want to ignore errors from the tasks when they run I think | 21:48 |
| clarkb | which we dont | 21:48 |
| dmsimard | Ah, right | 21:50 |
| dmsimard | I need to go back to "celebrate" Canada day (aka moving day in Quebec) but I'll be back tomorrow | 21:51 |
| clarkb | no worries, this isn't a rush. Was something I ran into (the warnings) when trinyg to follow logs and decided cleaning them up was worthwhile | 21:51 |
| ianw | clarkb: not totally caught up, did you see my note on the skip too? | 21:53 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Load extra Zuul config from zuul-jobs https://review.opendev.org/668563 | 21:54 |
| ianw | infra-root: also; https://review.opendev.org/#/c/667782/ is i think ready for review. it moves the rsync mirror updates to a opendev.org host managed by ansible. the idea is to get everything on it, but reprepro and all the gpg key handling etc will need some more extensive hacking. i figure it's a good start | 21:56 |
| clarkb | ianw: oh I completely missed htat | 21:56 |
| clarkb | ianw: so it is indeed a bug in ansible | 21:56 |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 21:57 |
| *** tdasilva_ has joined #openstack-infra | 21:57 | |
| *** weifan has quit IRC | 21:58 | |
| clarkb | ianw: any idea if ansible is backporting that fix? seems like an important one if they are going to deprecate that functionality with skip | 21:58 |
| ianw | clarkb: yeah, the skip bit is/was ... we could ask for a backport | 21:58 |
| ianw | jinx :) | 21:58 |
| ianw | pabelanger: ^ might have the contacts to get a backport in a (more) expedient fashion? | 21:59 |
| ianw | https://github.com/ansible/ansible/pull/57038 direct link | 21:59 |
| *** tdasilva has quit IRC | 22:00 | |
| openstackgerrit | Merged zuul/zuul master: Reload config on extra config files changes https://review.opendev.org/668548 | 22:00 |
| clarkb | also thanks for the update on the afs mirrors. Waiting on donnyd to get the cloud back into a happy state then I can deploy a mirror there with openafs 1.8.3 | 22:01 |
| clarkb | though may do some quick nova based perf testing first | 22:01 |
| donnyd | Still working | 22:01 |
| clarkb | donnyd: no rush | 22:02 |
| donnyd | This isn | 22:02 |
| donnyd | well I would like to move on to actually using it | 22:02 |
| donnyd | LOL | 22:02 |
| *** mattw4 has joined #openstack-infra | 22:03 | |
| clarkb | well I don't want you to feel you have to rush on our behalf, but yes we too are excited about using it :) | 22:03 |
| ianw | clarkb: cool. dhowells mentioned that cachefiles issues probably wouldn't get much time until maybe mid july. that seems to be the major stability issue | 22:04 |
| ianw | in the mean time if we can help out with details to get the vos release / kafs seeing servers offline possible race issue sorted, that's one less thing too | 22:04 |
| ianw | so we're certainly not in "set and forget" mode with kafs, but at least we're pointed in the right direction | 22:05 |
| *** weifan has joined #openstack-infra | 22:05 | |
| donnyd | I am not sure when NFS stopped working OOB.. but it surely is giving me quite a rough time today | 22:06 |
| donnyd | LOL | 22:06 |
| *** bobh has quit IRC | 22:06 | |
| corvus | clarkb: wow you are not kidding about the deprecation warnings | 22:08 |
| corvus | the logs are just useless | 22:08 |
| clarkb | corvus: yup :/ | 22:08 |
| *** weifan has quit IRC | 22:09 | |
| corvus | it's the same 6 line deprecation warning every 0.05 seconds. | 22:09 |
| corvus | i'm not sure we're ready for this yet -- but maybe in the near future we could set deprecation_warnings=False in production, but in zuul, have jobs report warnings back to zuul when deprecation warnings show up in runs... | 22:11 |
| corvus | (still, don't fail the jobs, just have them show up in the reports) | 22:12 |
| *** mriedem has quit IRC | 22:13 | |
| corvus | the extra config zuul changes have landed, i'm going to restart zuul | 22:13 |
| corvus | and the repl server is in, so no more cherry-picking required | 22:14 |
| clarkb | corvus: that is an interesting idea. I think as long as we expose those warnings that would be fine. Maybe a separate log file even | 22:14 |
| *** rh-jelabarre has quit IRC | 22:15 | |
| corvus | clarkb: yeah, i'm thinking something that ends up in the report message so we know to go looking for them -- but given that we need to identify them in order to know to raise the alarm, making a separate log file for them should be easy | 22:15 |
| *** rfarr_ has joined #openstack-infra | 22:16 | |
| *** hamzy has quit IRC | 22:18 | |
| *** rfarr has quit IRC | 22:19 | |
| *** pkopec has quit IRC | 22:22 | |
| *** rh-jelabarre has joined #openstack-infra | 22:22 | |
| *** weifan has joined #openstack-infra | 22:24 | |
| *** jamesmcarthur has quit IRC | 22:27 | |
| *** jamesmcarthur has joined #openstack-infra | 22:27 | |
| *** rfarr_ has quit IRC | 22:27 | |
| *** rfarr_ has joined #openstack-infra | 22:28 | |
| *** jamesmcarthur_ has joined #openstack-infra | 22:28 | |
| *** rcernin has joined #openstack-infra | 22:30 | |
| corvus | #status log restarted all of zuul on commit c5090244dc608a1ef232edded5cf92bf753dbb12 | 22:31 |
| openstackstatus | corvus: finished logging | 22:31 |
| *** jamesmcarthur has quit IRC | 22:32 | |
| *** tdasilva_ has quit IRC | 22:40 | |
| *** tdasilva_ has joined #openstack-infra | 22:41 | |
| *** rfarr_ has quit IRC | 22:44 | |
| *** smarcet has quit IRC | 22:44 | |
| *** rfarr_ has joined #openstack-infra | 22:44 | |
| clarkb | ianw: re the mirror update node change: is the concern with updating the reprepro stuff that it changes on eg bionic or that it will just be more effort to convert from puppet to ansible? | 22:46 |
| *** rfarr_ has quit IRC | 22:46 | |
| *** rfarr_ has joined #openstack-infra | 22:46 | |
| *** rfarr_ has quit IRC | 22:48 | |
| *** rfarr_ has joined #openstack-infra | 22:49 | |
| clarkb | also double checking I understand the testing there. It will deploy a mirror update node that will be set up to talk to the real afs volumes but the keytabs won't auth? | 22:50 |
| clarkb | so everything but the actual cron entry execution is exercised | 22:50 |
| corvus | yeah, my read is that its basically a test that the files got put on disk... | 22:51 |
| clarkb | ya left another note that we might want to also test cron was updated | 22:51 |
| clarkb | in any case +2'd it and if my understanding is correct feel free to approve | 22:52 |
| corvus | i think it would be okay to put in a real keytab for an account which didn't have any special access, but i don't think that gets us much farther, we still won't be able to run the real rsync and release commands | 22:52 |
| clarkb | ya | 22:52 |
| corvus | we could parameterize the location, but then having a second copy of all the data to rsync sounds pretty heavyweight. so all in all, i think what's there is about as much testing as is practical | 22:53 |
| *** aaronsheffield has quit IRC | 22:53 | |
| *** tkajinam has joined #openstack-infra | 22:53 | |
| corvus | (especially since this is a service where, when it breaks, everything just stays the same) | 22:54 |
| *** rlandy has quit IRC | 22:56 | |
| *** hamzy has joined #openstack-infra | 22:57 | |
| clarkb | firefox has decided it needs an entire cpu 100% of the time. So I'm gonna reboot and pick up some much needed updates. Back in a bit | 22:57 |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 22:58 |
| *** eernst has quit IRC | 22:59 | |
| *** hwoarang has quit IRC | 23:00 | |
| *** hwoarang has joined #openstack-infra | 23:01 | |
| *** eernst has joined #openstack-infra | 23:02 | |
| *** Lucas_Gray has quit IRC | 23:02 | |
| *** igordc has joined #openstack-infra | 23:03 | |
| *** eernst has quit IRC | 23:04 | |
| *** jamesmcarthur_ has quit IRC | 23:05 | |
| *** igordc has quit IRC | 23:05 | |
| *** igordc has joined #openstack-infra | 23:06 | |
| ianw | clarkb: yeah, no so much a concern as bigger job converting the reprepro puppet | 23:06 |
| ianw | yeah, the test is really just making sure things are decoded properly at this point, certainly could do more | 23:07 |
| *** tosky has quit IRC | 23:08 | |
| *** tdasilva has joined #openstack-infra | 23:10 | |
| *** tdasilva_ has quit IRC | 23:11 | |
| *** tdasilva has quit IRC | 23:11 | |
| *** tdasilva has joined #openstack-infra | 23:12 | |
| *** hwoarang has quit IRC | 23:14 | |
| *** hwoarang has joined #openstack-infra | 23:17 | |
| *** rh-jelabarre has quit IRC | 23:18 | |
| *** smarcet has joined #openstack-infra | 23:18 | |
| *** andreaf has quit IRC | 23:18 | |
| ianw | corvus: i think for 100% safety that should be fixed? but i think you have to be unlucky to get things that trigger an escape. | 23:20 |
| *** tdasilva has quit IRC | 23:20 | |
| ianw | i guess values in unicode code-points; maybe keytabs don't even have that | 23:21 |
| *** weifan has quit IRC | 23:22 | |
| *** tdasilva has joined #openstack-infra | 23:22 | |
| *** tdasilva has quit IRC | 23:22 | |
| *** tdasilva has joined #openstack-infra | 23:22 | |
| ianw | it seems like "ansible vault" is the answer ... not sure I want to ask the question though :) | 23:23 |
| clarkb | that seems half an answer | 23:23 |
| clarkb | there is such a thing as binary data that isn't secret | 23:23 |
| clarkb | might even be a constant | 23:23 |
| ianw | very true. i thought the idea of a "data:" analog to "content:" for copy might have been good. i find that many responses to ansible issues on github don't exactly foster discussion | 23:26 |
| *** dchen has joined #openstack-infra | 23:32 | |
| *** andreaf has joined #openstack-infra | 23:34 | |
| *** rfarr_ has quit IRC | 23:38 | |
| *** rfarr_ has joined #openstack-infra | 23:39 | |
| *** sthussey has quit IRC | 23:39 | |
| *** rfarr_ has quit IRC | 23:50 | |
| *** rfarr_ has joined #openstack-infra | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!