| *** smarcet has left #openstack-infra | 00:02 | |
| *** yamamoto has joined #openstack-infra | 00:04 | |
| *** calbers has quit IRC | 00:04 | |
| corvus | ianw: +2 with comment (sorry if that's pedantic) | 00:06 |
|---|---|---|
| corvus | i also approved the dns change cause it's just adding a new host | 00:06 |
| *** rfarr_ has quit IRC | 00:07 | |
| *** rfarr_ has joined #openstack-infra | 00:07 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add mirror-update01.opendev.org https://review.opendev.org/668610 | 00:08 |
| *** jamesmcarthur has joined #openstack-infra | 00:08 | |
| ianw | corvus: thanks; np, i probably should have said "the mirror volume keytabs" or something more clear | 00:11 |
| corvus | yeah, the meaning entirely changes with something like that or the word "this" :) | 00:12 |
| corvus | i figured that's what you meant, but just in case | 00:12 |
| *** calbers has joined #openstack-infra | 00:13 | |
| *** slaweq has joined #openstack-infra | 00:16 | |
| *** rfarr_ has quit IRC | 00:17 | |
| *** rfarr_ has joined #openstack-infra | 00:17 | |
| *** jamesmcarthur has quit IRC | 00:19 | |
| openstackgerrit | Merged opendev/puppet-mediawiki master: Canonicalize clone URLs https://review.opendev.org/666162 | 00:19 |
| ianw | i was also poking around a bit with mount namespaces and containers etc for kafs. i think it all means good things for containerisation ... although the point is a bit moot right now when you need custom kernels on the host so it's not exactly "deploy anywhere", but the future is bright :) | 00:20 |
| *** slaweq has quit IRC | 00:21 | |
| ianw | corvus / clarkb: also, for phase 2 when i get the jobs working on the new host; my idea was to write another cron job to rsync the mirroring logs to a separate volume and just expose that. i don't think logging directly to the volume is a good idea, because all the jobs kick off at random times and we couldn't really know when is a good, non-racy time to vos release | 00:22 |
| corvus | ianw: we could log directly to a rw volume | 00:23 |
| ianw | and just not replicate it? | 00:25 |
| *** ijw has joined #openstack-infra | 00:26 | |
| *** calbers has quit IRC | 00:26 | |
| *** calbers has joined #openstack-infra | 00:27 | |
| *** gyee has quit IRC | 00:28 | |
| openstackgerrit | Merged opendev/system-config master: Apply service-bridge.yaml in run_all.sh https://review.opendev.org/668762 | 00:28 |
| *** rfarr_ has quit IRC | 00:32 | |
| *** rfarr_ has joined #openstack-infra | 00:33 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 00:33 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 00:34 |
| corvus | ianw: yep. it just means they won't be available if that fileserver goes offline | 00:35 |
| corvus | that's probably okay for those logs? | 00:35 |
| ianw | yep; i'll have to think about authentication ... currently the logs are just > file from the cron job ... that could be fixed ... logrotate though, i'm not sure | 00:43 |
| openstackgerrit | Merged opendev/system-config master: Move rsync mirror updates to new opendev.org mirror-update host https://review.opendev.org/667782 | 00:44 |
| openstackgerrit | Merged opendev/system-config master: mirror-update: update keytab testing https://review.opendev.org/668620 | 00:44 |
| openstackgerrit | Merged opendev/system-config master: Add mirror-update01.opendev.org server https://review.opendev.org/668612 | 00:44 |
| *** Lucas_Gray has quit IRC | 00:44 | |
| donnyd | clarkb: Is there anything I can do to build the mirror node? | 00:44 |
| clarkb | donnyd: no we need to run our launch node scripy and enroll it into our inventory | 00:44 |
| clarkb | which should work soonish now that the fix has merged | 00:45 |
| donnyd | Ok cool | 00:45 |
| *** ijw has quit IRC | 00:47 | |
| *** ijw has joined #openstack-infra | 00:49 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base integration roles https://review.opendev.org/668061 | 00:49 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 00:49 |
| *** rfarr_ has quit IRC | 00:51 | |
| *** rfarr_ has joined #openstack-infra | 00:51 | |
| *** ricolin_ has joined #openstack-infra | 00:51 | |
| donnyd | spinning up an instance in openstackci just to check that its all working | 00:52 |
| *** ricolin_ is now known as ricolin | 00:52 | |
| donnyd | and you needed two floating ips for openstackci? correct clarkb ? | 00:53 |
| clarkb | donnyd: one for the mirror then another ip that acts as nat for the other nodes | 00:55 |
| clarkb | the second one jsnt a floating ip and may just use an already in use IP | 00:55 |
| clarkb | ? | 00:55 |
| donnyd | outbound nat? | 00:55 |
| donnyd | or for inbound traffic? | 00:56 |
| donnyd | outbound is handled by the openstack router | 00:56 |
| donnyd | which is outbound nat by default | 00:56 |
| *** ramishra has quit IRC | 00:56 | |
| clarkb | ya outbound so thats handled then | 00:58 |
| donnyd | Anyway, I am just testing this project to make sure traffic is flowing and then I am going to head out for the night | 00:59 |
| clarkb | services like dockerhub and github dont ipv6 so we need that and we cache what we can too | 00:59 |
| clarkb | ok thanks for all the help! | 00:59 |
| *** yamamoto has quit IRC | 00:59 | |
| *** yamamoto has joined #openstack-infra | 00:59 | |
| donnyd | so the mirror node needs 1 ip and then everything else needs outbound nat to reach the githubs and what not | 00:59 |
| donnyd | ok, both projects are good to go. | 01:00 |
| *** panda has quit IRC | 01:03 | |
| *** panda has joined #openstack-infra | 01:05 | |
| *** jklare has quit IRC | 01:08 | |
| clarkb | yup | 01:10 |
| *** imacdonn has quit IRC | 01:11 | |
| *** imacdonn has joined #openstack-infra | 01:11 | |
| *** smarcet has joined #openstack-infra | 01:13 | |
| *** jklare has joined #openstack-infra | 01:21 | |
| ianw | #status log post https://review.opendev.org/667782 rsync cron jobs commented out on mirror-update.openstack.org | 01:25 |
| openstackstatus | ianw: finished logging | 01:25 |
| *** jklare has quit IRC | 01:29 | |
| *** jklare has joined #openstack-infra | 01:29 | |
| *** yamamoto has quit IRC | 01:30 | |
| *** rfarr_ has quit IRC | 01:32 | |
| *** rfarr_ has joined #openstack-infra | 01:33 | |
| ianw | ahh, i didn't inject enough randomness into the cron job start times | 01:34 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: seed cron time with unique job name https://review.opendev.org/668768 | 01:38 |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: ironic-agent: Use targetcli & python3-devel on rhel8 https://review.opendev.org/668687 | 01:39 |
| *** rfarr_ has quit IRC | 01:40 | |
| *** yamamoto has joined #openstack-infra | 01:40 | |
| *** rfarr_ has joined #openstack-infra | 01:40 | |
| *** rfarr_ has quit IRC | 01:47 | |
| *** rfarr_ has joined #openstack-infra | 01:48 | |
| *** sgw has quit IRC | 01:51 | |
| *** yamamoto has quit IRC | 01:51 | |
| *** bobh has joined #openstack-infra | 01:53 | |
| *** apetrich has quit IRC | 01:57 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/668772 | 02:07 |
| smarcet | fungi: please review at will ^^^^ | 02:08 |
| *** ijw_ has joined #openstack-infra | 02:14 | |
| *** yamamoto has joined #openstack-infra | 02:15 | |
| *** ijw has quit IRC | 02:17 | |
| *** jcoufal has quit IRC | 02:18 | |
| *** rfarr_ has quit IRC | 02:20 | |
| *** rfarr_ has joined #openstack-infra | 02:21 | |
| *** ijw_ has quit IRC | 02:24 | |
| *** ijw has joined #openstack-infra | 02:24 | |
| *** rfarr_ has quit IRC | 02:26 | |
| *** rfarr_ has joined #openstack-infra | 02:26 | |
| *** ijw has quit IRC | 02:29 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:34 | |
| *** rlandy has quit IRC | 02:34 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:36 | |
| *** yamamoto has quit IRC | 02:36 | |
| *** bhavikdbavishi has quit IRC | 02:38 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:38 | |
| *** bobh has quit IRC | 02:47 | |
| *** rfolco has quit IRC | 02:47 | |
| *** rfarr_ has quit IRC | 02:54 | |
| *** rfarr_ has joined #openstack-infra | 02:55 | |
| *** xinranwang has joined #openstack-infra | 03:04 | |
| *** rfarr_ has quit IRC | 03:07 | |
| *** rfarr_ has joined #openstack-infra | 03:08 | |
| *** StevenK has quit IRC | 03:08 | |
| *** yamamoto has joined #openstack-infra | 03:15 | |
| *** smarcet has left #openstack-infra | 03:22 | |
| *** zzehring has quit IRC | 03:23 | |
| *** yamamoto has quit IRC | 03:25 | |
| *** zzehring has joined #openstack-infra | 03:26 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: fix cron path https://review.opendev.org/668780 | 03:33 |
| *** psachin has joined #openstack-infra | 03:34 | |
| *** StevenK has joined #openstack-infra | 03:37 | |
| *** ramishra has joined #openstack-infra | 03:42 | |
| *** rfarr_ has quit IRC | 03:49 | |
| *** rfarr_ has joined #openstack-infra | 03:49 | |
| *** rfarr_ has quit IRC | 03:52 | |
| *** rfarr_ has joined #openstack-infra | 03:53 | |
| *** ricolin_ has joined #openstack-infra | 03:54 | |
| *** udesale has joined #openstack-infra | 03:56 | |
| *** ricolin has quit IRC | 03:57 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/668772 | 03:59 |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/668772 | 04:00 |
| *** ykarel|away has joined #openstack-infra | 04:01 | |
| *** ykarel|away is now known as ykarel | 04:06 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/668772 | 04:06 |
| *** jamesmcarthur has joined #openstack-infra | 04:08 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/#/c/661661/ https://review.opendev.org/668772 | 04:14 |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/#/c/661661/ https://review.opendev.org/668772 | 04:16 |
| *** ykarel has quit IRC | 04:17 | |
| openstackgerrit | Merged opendev/system-config master: mirror-update: seed cron time with unique job name https://review.opendev.org/668768 | 04:24 |
| openstackgerrit | Merged opendev/system-config master: mirror-update: fix cron path https://review.opendev.org/668780 | 04:24 |
| *** jamesmcarthur has quit IRC | 04:24 | |
| *** rfarr_ has quit IRC | 04:25 | |
| *** rfarr_ has joined #openstack-infra | 04:26 | |
| *** toabctl has quit IRC | 04:31 | |
| *** whoami-rajat has joined #openstack-infra | 04:34 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/#/c/661661/ https://review.opendev.org/668772 | 04:34 |
| *** kjackal has joined #openstack-infra | 04:35 | |
| *** ykarel has joined #openstack-infra | 04:38 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/#/c/661661/ https://review.opendev.org/668772 | 04:43 |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/#/c/661661/ https://review.opendev.org/668772 | 04:55 |
| *** dychen has joined #openstack-infra | 04:56 | |
| *** dychen has quit IRC | 04:58 | |
| *** dchen has quit IRC | 04:58 | |
| *** dchen has joined #openstack-infra | 05:01 | |
| *** tkajinam has quit IRC | 05:01 | |
| *** pcaruana has joined #openstack-infra | 05:06 | |
| *** rfarr_ has quit IRC | 05:08 | |
| *** rfarr_ has joined #openstack-infra | 05:08 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files and bc this , checking of this commit is failing https://review.opendev.org/#/c/661661/ https://review.opendev.org/668772 | 05:14 |
| *** e0ne has joined #openstack-infra | 05:20 | |
| *** rfarr_ has quit IRC | 05:22 | |
| *** rfarr_ has joined #openstack-infra | 05:22 | |
| *** kjackal has quit IRC | 05:30 | |
| *** ykarel_ has joined #openstack-infra | 05:30 | |
| *** ykarel has quit IRC | 05:32 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files. Added MYSQL dependencies in order to run unit IDP unit test. Added php unit invocations for main tests * OIDC * OAUTH2 * OPENID https://review.opendev.org/668772 | 05:33 |
| *** ricolin_ is now known as ricolin | 05:37 | |
| *** ijw has joined #openstack-infra | 05:37 | |
| *** udesale has quit IRC | 05:38 | |
| *** ijw_ has joined #openstack-infra | 05:38 | |
| *** rfarr_ has quit IRC | 05:39 | |
| *** rfarr_ has joined #openstack-infra | 05:39 | |
| *** igordc has quit IRC | 05:41 | |
| *** ykarel_ has quit IRC | 05:41 | |
| *** tdasilva_ has joined #openstack-infra | 05:41 | |
| *** ijw has quit IRC | 05:42 | |
| *** rfarr_ has quit IRC | 05:43 | |
| *** rfarr_ has joined #openstack-infra | 05:43 | |
| *** ykarel has joined #openstack-infra | 05:44 | |
| *** tdasilva has quit IRC | 05:44 | |
| *** rfarr_ has quit IRC | 05:49 | |
| *** rfarr_ has joined #openstack-infra | 05:49 | |
| *** rfarr_ has quit IRC | 05:58 | |
| *** rfarr_ has joined #openstack-infra | 05:58 | |
| *** tkajinam has joined #openstack-infra | 06:00 | |
| yoctozepto | > <ianw> yoctozepto / clarkb: ^ i really do like our testing finding issues that hadn't thought of :) although i should have put a comment first time around because i remember now why it was done like that | 06:06 |
| yoctozepto | yeah, it always helps | 06:06 |
| yoctozepto | always learning the hard way ;D | 06:06 |
| *** ramishra has quit IRC | 06:08 | |
| *** kjackal has joined #openstack-infra | 06:11 | |
| *** e0ne has quit IRC | 06:12 | |
| *** rfarr_ has quit IRC | 06:15 | |
| *** rfarr_ has joined #openstack-infra | 06:16 | |
| *** ramishra has joined #openstack-infra | 06:16 | |
| *** snapiri has quit IRC | 06:26 | |
| *** snapiri has joined #openstack-infra | 06:26 | |
| *** rfarr_ has quit IRC | 06:30 | |
| *** rfarr_ has joined #openstack-infra | 06:30 | |
| *** rfarr_ has quit IRC | 06:34 | |
| *** rfarr_ has joined #openstack-infra | 06:34 | |
| *** witek has joined #openstack-infra | 06:36 | |
| *** apetrich has joined #openstack-infra | 06:42 | |
| *** rfarr_ has quit IRC | 06:43 | |
| *** rfarr_ has joined #openstack-infra | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:46 | |
| *** dychen has joined #openstack-infra | 06:49 | |
| *** ricolin_ has joined #openstack-infra | 06:51 | |
| *** rfarr_ has quit IRC | 06:51 | |
| *** ijw_ has quit IRC | 06:52 | |
| *** rfarr_ has joined #openstack-infra | 06:52 | |
| *** dchen has quit IRC | 06:52 | |
| *** ricolin has quit IRC | 06:53 | |
| *** dychen has quit IRC | 06:54 | |
| *** dychen has joined #openstack-infra | 06:54 | |
| *** dychen has quit IRC | 07:00 | |
| *** dchen has joined #openstack-infra | 07:01 | |
| *** ginopc has joined #openstack-infra | 07:05 | |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| *** xek has joined #openstack-infra | 07:07 | |
| *** rfarr_ has quit IRC | 07:09 | |
| *** rfarr_ has joined #openstack-infra | 07:10 | |
| *** tdasilva_ has quit IRC | 07:14 | |
| *** tdasilva_ has joined #openstack-infra | 07:14 | |
| *** dychen has joined #openstack-infra | 07:15 | |
| *** dchen has quit IRC | 07:15 | |
| *** pgaxatte has joined #openstack-infra | 07:15 | |
| *** dingyichen has joined #openstack-infra | 07:16 | |
| *** dychen has quit IRC | 07:19 | |
| *** tosky has joined #openstack-infra | 07:20 | |
| *** tesseract has joined #openstack-infra | 07:21 | |
| *** xinranwang has quit IRC | 07:24 | |
| *** rfarr_ has quit IRC | 07:25 | |
| *** rfarr_ has joined #openstack-infra | 07:26 | |
| *** toabctl has joined #openstack-infra | 07:30 | |
| *** rfarr_ has quit IRC | 07:36 | |
| *** rfarr_ has joined #openstack-infra | 07:37 | |
| *** rfarr_ has quit IRC | 07:53 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** rfarr_ has joined #openstack-infra | 07:54 | |
| *** ralonsoh has joined #openstack-infra | 07:59 | |
| *** jamesmcarthur has joined #openstack-infra | 08:00 | |
| *** ociuhandu has joined #openstack-infra | 08:05 | |
| *** ociuhandu has quit IRC | 08:05 | |
| *** jamesmcarthur has quit IRC | 08:06 | |
| *** ociuhandu has joined #openstack-infra | 08:06 | |
| *** trident has quit IRC | 08:08 | |
| *** trident has joined #openstack-infra | 08:09 | |
| *** dingyichen has quit IRC | 08:10 | |
| *** ykarel has quit IRC | 08:13 | |
| *** tdasilva_ has quit IRC | 08:14 | |
| *** ykarel has joined #openstack-infra | 08:15 | |
| *** ykarel is now known as ykarel|lunch | 08:15 | |
| *** pkopec has joined #openstack-infra | 08:22 | |
| *** priteau has joined #openstack-infra | 08:26 | |
| *** lucasagomes has joined #openstack-infra | 08:28 | |
| *** rfarr_ has quit IRC | 08:30 | |
| *** rfarr_ has joined #openstack-infra | 08:30 | |
| *** iurygregory has joined #openstack-infra | 08:32 | |
| openstackgerrit | Merged opendev/irc-meetings master: Adjust Airship meeting time https://review.opendev.org/668755 | 08:35 |
| *** ykarel has joined #openstack-infra | 08:36 | |
| *** ykarel|lunch has quit IRC | 08:36 | |
| yoctozepto | infra, another case for job run twice (or more) internally: http://logs.openstack.org/15/668815/1/check/kolla-ansible-centos-source/756113e/ | 08:38 |
| *** tkajinam has quit IRC | 08:39 | |
| yoctozepto | ara and main logs show that they failed due to file already existing, yet primary node logs show that the job was proceeding further just fine | 08:39 |
| *** ricolin__ has joined #openstack-infra | 08:44 | |
| *** rfarr_ has quit IRC | 08:44 | |
| *** rfarr_ has joined #openstack-infra | 08:45 | |
| *** ricolin_ has quit IRC | 08:47 | |
| *** zzzeek has quit IRC | 08:48 | |
| *** Fidde has joined #openstack-infra | 08:48 | |
| *** zzzeek has joined #openstack-infra | 08:49 | |
| *** ykarel_ has joined #openstack-infra | 08:51 | |
| *** ykarel has quit IRC | 08:54 | |
| *** ricolin__ is now known as ricolin | 08:55 | |
| *** tesseract-RH has joined #openstack-infra | 08:57 | |
| *** tesseract has quit IRC | 08:58 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: export mirroring logs https://review.opendev.org/668832 | 09:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: update docs for mirror-update.opendev.org https://review.opendev.org/668833 | 09:02 |
| *** Lucas_Gray has joined #openstack-infra | 09:04 | |
| *** udesale has joined #openstack-infra | 09:04 | |
| *** tdasilva has joined #openstack-infra | 09:08 | |
| *** rfarr_ has quit IRC | 09:11 | |
| *** rfarr_ has joined #openstack-infra | 09:11 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 09:19 | |
| *** Fidde has quit IRC | 09:22 | |
| *** lucasagomes has quit IRC | 09:26 | |
| *** lucasagomes has joined #openstack-infra | 09:26 | |
| sshnaidm|ruck | I see rax mirror problem today: http://logs.openstack.org/17/668717/1/gate/puppet-openstack-lint-ubuntu-bionic/4556fe9/job-output.txt.gz#_2019-07-02_19_37_26_081163 | 09:26 |
| *** lucasagomes has quit IRC | 09:27 | |
| *** ykarel_ is now known as ykarel | 09:27 | |
| *** tesseract-RH has quit IRC | 09:29 | |
| *** tesseract has joined #openstack-infra | 09:29 | |
| *** lucasagomes has joined #openstack-infra | 09:30 | |
| *** tesseract has quit IRC | 09:33 | |
| *** tesseract has joined #openstack-infra | 09:33 | |
| *** rfarr_ has quit IRC | 09:36 | |
| *** rfarr_ has joined #openstack-infra | 09:36 | |
| *** e0ne has joined #openstack-infra | 09:37 | |
| roman_g | Hello team. Is it a temporary failure, or some issue? -> Could not connect to mirror.iad.rax.opendev.org:80 (2001:4802:7802:104:be76:4eff:fe20:4b35), connection timed out Could not connect to mirror.iad.rax.opendev.org:80 (104.130.4.160), connection timed out | 09:38 |
| *** tesseract has quit IRC | 09:38 | |
| *** tosky has quit IRC | 09:39 | |
| *** tesseract has joined #openstack-infra | 09:39 | |
| *** tosky has joined #openstack-infra | 09:40 | |
| yoctozepto | sshnaidm|ruck: it was yesterday, should be solved by now | 09:41 |
| yoctozepto | roman_g: seems iad.rax is down ;-( | 09:41 |
| sshnaidm|ruck | is it cloudfare to blame? :) | 09:42 |
| *** bhavikdbavishi has quit IRC | 09:42 | |
| yoctozepto | ping ianw, fungi ^ | 09:42 |
| yoctozepto | sshnaidm|ruck: no, openafs client had issues | 09:42 |
| yoctozepto | though cloudflare showed how much of internet is under them ;D | 09:43 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Initiate retirement of openstack/docs-specs https://review.opendev.org/668853 | 09:56 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Retire docs-specs https://review.opendev.org/668854 | 09:56 |
| *** ginopc has quit IRC | 10:00 | |
| *** ginopc has joined #openstack-infra | 10:00 | |
| roman_g | primary | Get:14 http://mirror.ord.rax.opendev.org/ubuntu bionic/main amd64 Packages [1547 kB] | 10:01 |
| roman_g | ^^^ sometimes it works | 10:01 |
| yoctozepto | roman_g: it's ord.rax | 10:01 |
| roman_g | ah. | 10:01 |
| roman_g | true, thanks | 10:01 |
| *** rfarr_ has quit IRC | 10:01 | |
| yoctozepto | iad.rax uses kafs which is experimental | 10:01 |
| *** lei-zh has joined #openstack-infra | 10:02 | |
| yoctozepto | though ord.rax failed just yesterday | 10:02 |
| yoctozepto | so not as stable as expected | 10:02 |
| roman_g | kafs? what is that? | 10:02 |
| *** rfarr_ has joined #openstack-infra | 10:02 | |
| yoctozepto | kernel client for afs | 10:02 |
| *** Lucas_Gray has quit IRC | 10:07 | |
| lei-zh | hello, does anyone know how gerrit merges are pushed to github, where to configure this | 10:08 |
| *** dmsimard9 has joined #openstack-infra | 10:08 | |
| lei-zh | we have an openstack project stop syncing changes to github for months | 10:08 |
| *** dmsimard has quit IRC | 10:08 | |
| *** ginopc has quit IRC | 10:09 | |
| *** Lucas_Gray has joined #openstack-infra | 10:09 | |
| roman_g | lei-zh: https://opendev.org/airship/armada/src/branch/master/.zuul.yaml#L206-L215 this is how we do it | 10:10 |
| roman_g | lei-zh: your case might be different, depending on your repository name (url path), as we are not under openstack/ | 10:11 |
| lei-zh | roman_g: thanks mate | 10:12 |
| roman_g | lei-zh: always welcome | 10:12 |
| roman_g | Failed to fetch http://mirror.iad.rax.opendev.org/ubuntu/dists/bionic/InRelease Could not connect to mirror.iad.rax.opendev.org:80 (2001:4802:7802:104:be76:4eff:fe20:4b35). - connect (113: No route to host) Could not connect to mirror.iad.rax.opendev.org:80 (104.130.4.160). - connect (113: No route to host) | 10:16 |
| roman_g | something new | 10:16 |
| roman_g | But what is nice is that job gets re-scheduled again. Don't know how. | 10:17 |
| frickler | infra-root: mirror01.iad.rax.opendev.org is being shown as SHUTOFF in the cli. I'm going to try and start it again | 10:18 |
| *** rfarr_ has quit IRC | 10:20 | |
| *** rfarr_ has joined #openstack-infra | 10:21 | |
| ianw | frickler: yeah, we caught a panic | 10:25 |
| frickler | seems that what happened a week ago already, server is now shown as active again. I'm getting a 403 though on http://mirror.iad.rax.opendev.org/ubuntu/dists/bionic/InRelease | 10:25 |
| *** ykarel_ has joined #openstack-infra | 10:25 | |
| frickler | ianw: I read something about a netlog server, which one is that? | 10:26 |
| ianw | frickler: details are in http://etherpad.openstack.org/opendev-mirror-afs | 10:27 |
| *** ykarel has quit IRC | 10:27 | |
| ianw | frickler: i've just run service kafs start on iad, that should get /afs mounted again | 10:28 |
| *** ginopc has joined #openstack-infra | 10:28 | |
| *** udesale has quit IRC | 10:28 | |
| *** udesale has joined #openstack-infra | 10:29 | |
| ianw | oh damn it i might have just accidentally rebooted it | 10:29 |
| ianw | sorry i did | 10:29 |
| *** ykarel has joined #openstack-infra | 10:30 | |
| frickler | ianw: I had restarted apache2 in parallel, but it still works after the reboot now, too | 10:30 |
| *** ykarel_ has quit IRC | 10:30 | |
| ianw | yep, it's just getting kafs started. i've also restarted the netconsole, but the oops we got is the same filecache one | 10:31 |
| ianw | i'm going to have to sync with dhowells on this; if we can work around it, we'll either have to turn off filecache or turn of kafs for a while till there's a solution | 10:31 |
| *** kjackal has quit IRC | 10:33 | |
| *** ykarel is now known as ykarel|meeting | 10:33 | |
| *** lei-zh has quit IRC | 10:36 | |
| *** rfarr_ has quit IRC | 10:36 | |
| *** rfarr_ has joined #openstack-infra | 10:37 | |
| *** udesale has quit IRC | 10:52 | |
| ianw | from discussions with dhowells, he's going to try and spin up some custom stuff to dump more helpful info we can incorporate. we can collect that and we may have to just switch back in the openafs servers for a bit while we evaluate the details | 10:52 |
| *** ykarel|meeting is now known as ykarel | 10:54 | |
| ianw | will depend on if things become clearer or foggier :) | 10:54 |
| *** udesale has joined #openstack-infra | 10:54 | |
| *** udesale has quit IRC | 10:58 | |
| *** udesale has joined #openstack-infra | 10:58 | |
| *** kjackal has joined #openstack-infra | 11:01 | |
| sshnaidm|ruck | Is is worth to use fail-fast in gate queue now? | 11:02 |
| sshnaidm|ruck | gates jobs fail and just waste time and resources | 11:02 |
| *** psachin has quit IRC | 11:06 | |
| *** rfarr_ has quit IRC | 11:11 | |
| *** brett-soric has joined #openstack-infra | 11:11 | |
| *** rfarr_ has joined #openstack-infra | 11:12 | |
| *** smarcet has joined #openstack-infra | 11:12 | |
| *** smarcet has quit IRC | 11:15 | |
| *** udesale has quit IRC | 11:19 | |
| *** priteau has quit IRC | 11:32 | |
| *** iurygregory has quit IRC | 11:35 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Shard project-pipeline config https://review.opendev.org/668763 | 11:46 |
| *** ykarel has quit IRC | 11:51 | |
| openstackgerrit | Roman Gorshunov proposed opendev/system-config master: Mailing list for Airship https://review.opendev.org/666606 | 11:51 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add base integration roles https://review.opendev.org/668061 | 11:56 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 11:56 |
| smcginnis | Anyone around (other than Andreas) that can take a look at https://review.opendev.org/#/c/668466/ | 11:58 |
| smcginnis | It would be appreciated to help move along a Glance goal patch. | 11:58 |
| *** ekultails has joined #openstack-infra | 12:00 | |
| *** brett-soric has left #openstack-infra | 12:00 | |
| frickler | smcginnis: +3 | 12:07 |
| mordred | smcginnis: do I want to know why we need to set python_version when the base is -py37? or should I just go back to morning coffee? | 12:07 |
| AJaeger | mordred: to install the interpreter? | 12:08 |
| mordred | AJaeger: ah - thanks - my brain was reading it wrong | 12:09 |
| mordred | AJaeger: for some reason I thought -functional-py37 was the _parent_ ... so it seemed weird to me - I was thinking "we should set that variable in -functional-py37" :) | 12:10 |
| AJaeger | ah | 12:10 |
| *** elod has quit IRC | 12:11 | |
| smcginnis | frickler, mordred: THanks | 12:11 |
| AJaeger | config-core, please review new ironic project https://review.opendev.org/#/c/665321/ | 12:12 |
| smcginnis | mordred: Yeah, oversight on my part setting that up and not thinking to set the interpreter on the parent job. | 12:12 |
| AJaeger | smcginnis: nobody around to +2A your change https://review.opendev.org/#/c/668041/ ? | 12:13 |
| mordred | AJaeger, smcginnis: so the answer is clearly "go back to morning coffee" :) | 12:13 |
| AJaeger | ;) | 12:13 |
| smcginnis | mordred: Haha, I guess so! | 12:13 |
| smcginnis | AJaeger: Not sure why that one has taken so long. ;) | 12:13 |
| smcginnis | I suppose I can push my own through, but was hoping not to have to do that. | 12:13 |
| *** bhavikdbavishi has joined #openstack-infra | 12:13 | |
| AJaeger | ttx, could you +2A https://review.opendev.org/#/c/668041/ , please? | 12:14 |
| smcginnis | Maybe ttx might be willing to though? | 12:14 |
| AJaeger | yep, let's try to ping him ;) | 12:14 |
| *** rfarr__ has joined #openstack-infra | 12:14 | |
| mordred | I can +2 in that one for some reason | 12:14 |
| mordred | want me to +A it? | 12:14 |
| smcginnis | Sure, I think that should be safe. Two release team members are OK with at least counting me submitting it and ttx giving +2. | 12:15 |
| AJaeger | mordred: infra-core is in the group ;) | 12:15 |
| mordred | ah - so it's legit for me to vote on it even :) | 12:15 |
| AJaeger | thanks, mordred | 12:15 |
| smcginnis | Nice, my open patch queue is shrinking this morning. :) | 12:16 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Set python_version: 3.7 for openstack-tox-functional-py37 https://review.opendev.org/668466 | 12:16 |
| AJaeger | smcginnis: do you have a change up for governance repo for the retirement? | 12:16 |
| *** rfarr_ has quit IRC | 12:16 | |
| smcginnis | Yeah, one moment... | 12:17 |
| smcginnis | https://review.opendev.org/668047 | 12:17 |
| *** rfarr_ has joined #openstack-infra | 12:17 | |
| ttx | on it | 12:17 |
| *** priteau has joined #openstack-infra | 12:17 | |
| ttx | mordred beat me to it | 12:17 |
| AJaeger | ttx, mordred beat you ;) | 12:17 |
| * mordred does a dance | 12:18 | |
| *** rfarr__ has quit IRC | 12:19 | |
| *** rfarr__ has joined #openstack-infra | 12:22 | |
| *** rfarr_ has quit IRC | 12:24 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 12:28 |
| openstackgerrit | Merged openstack/project-config master: Moving ironic-prometheus-exporter under Ironic https://review.opendev.org/665321 | 12:29 |
| openstackgerrit | Merged openstack/project-config master: Retire the release-schedule-generator project https://review.opendev.org/668045 | 12:32 |
| *** rlandy has joined #openstack-infra | 12:33 | |
| *** _erlon_ has joined #openstack-infra | 12:33 | |
| AJaeger | mordred: if you fancy another review, I hope https://review.opendev.org/#/c/668061 will pass ;) That'S the next change in the zuul-jobs testing change series. | 12:34 |
| *** eharney has joined #openstack-infra | 12:37 | |
| *** haleyb is now known as haleyb|away | 12:40 | |
| *** psachin has joined #openstack-infra | 12:43 | |
| *** dmsimard9 is now known as dmsimard | 12:43 | |
| *** sthussey has joined #openstack-infra | 12:45 | |
| *** happyhemant has joined #openstack-infra | 12:46 | |
| dtroyer | So we (StarlingX) just noticed the OpenStack URLs in the Gerrit first-time-contributor messages. Is that function part of Gerrit itself or is it a bot we can teach to have context-aware responses? | 12:47 |
| *** elod has joined #openstack-infra | 12:48 | |
| fungi | dtroyer: i'm not really here but i already mentioned to someone else (i forget who) some weeks back that the hook script gerrit calls is passed project information so it should in theory be possible to add a lookup from something in the gerrit/projects.yaml in openstack/project-config where the message is added in opendev/jeepyb | 12:49 |
| dtroyer | fungi: ok, thanks for the pointer. I (or someone) will have a deeper look to see what it would take. I had no idea where to start... | 12:50 |
| fungi | dtroyer: https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/welcome_message.py | 12:50 |
| fungi | probably the hard part will be deciding how to track the mapping of a project to a particular community | 12:51 |
| fungi | or maybe an easier alternative is to make it more generic | 12:51 |
| dtroyer | I was thinking of just using the project namespaces with a default of the current mesage... | 12:52 |
| dtroyer | so it is purely opt-in | 12:52 |
| fungi | that could be doable. as i said, the script *does* know the project name (in theory) so could parse out the namespace | 12:52 |
| dtroyer | looks like the fun part will be tsting it :) | 12:54 |
| fungi | when we eventually get to realizing our plan from the ptg about having meta repos in each namespace, maybe non-default welcome messages can be provided in those | 12:54 |
| mordred | ++ | 12:54 |
| * mordred was just about to say the same thing as fungi | 12:55 | |
| * dtroyer is looking forward to our new meta repo overlords | 12:55 | |
| dtroyer | to beat mordred to the line :) | 12:55 |
| fungi | in the meantime, some extra logic in that script should provide a reasonable compromise | 12:55 |
| mordred | dtroyer: I think they'll solve all of our problems and we'll be able to kick back and stop doing work completely | 12:55 |
| * fungi has already stopped doing work completely | 12:56 | |
| dtroyer | so thinking a bit, I think I am OK with not altering the query and leaving it global to the whole Gerrit DB | 12:56 |
| dtroyer | only changing the post_message() bits | 12:56 |
| fungi | yeah, even just genericizing that message seems reasonable to me | 12:57 |
| dtroyer | the only real concern I heard was the URLs, the conent itself seems fine | 12:57 |
| *** elod_ has joined #openstack-infra | 12:58 | |
| *** elod has quit IRC | 12:58 | |
| dtroyer | thanks guys, I'll punt it up to some folk who are not about to take a couple of days off | 12:59 |
| *** jcoufal has joined #openstack-infra | 12:59 | |
| *** elod_ is now known as elod | 12:59 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 13:00 | |
| *** bhavikdbavishi has quit IRC | 13:02 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 13:02 | |
| clarkb | sshnaidm|ruck: I'm really not here yet, but fail fast in the gate wont help much. The real cost is in resetting all jobs for changes after the failure | 13:02 |
| *** udesale has joined #openstack-infra | 13:03 | |
| clarkb | sshnaidm|ruck: this is why I keep pushing for people to make jobs more reliable and quicker instead of fine tuning fast fail or sets of jobs that run in succession | 13:03 |
| clarkb | because our biggest costs often come due to failures | 13:03 |
| sshnaidm|ruck | clarkb, well, I thought if most of failures because of infrastructure it may help, it's just matter of luck there | 13:05 |
| *** elod has quit IRC | 13:06 | |
| clarkb | sshnaidm|ruck: it completely ignores the real cost of a failure in thr gate | 13:06 |
| clarkb | the cost isnt in completing the jobs for the change that failed, it is in resetting all of the jobs for changes later in the queue | 13:06 |
| clarkb | the impact of fail fast eould be tiny compared to improving the reliability of our testing | 13:07 |
| sshnaidm|ruck | clarkb, I see | 13:07 |
| *** elod has joined #openstack-infra | 13:07 | |
| *** nicolasbock has joined #openstack-infra | 13:11 | |
| *** ykarel has joined #openstack-infra | 13:12 | |
| *** owalsh has quit IRC | 13:13 | |
| *** owalsh has joined #openstack-infra | 13:14 | |
| sshnaidm|ruck | who can remove my twin from storyboard.openstack.org? | 13:14 |
| *** ijw has joined #openstack-infra | 13:15 | |
| *** owalsh has quit IRC | 13:15 | |
| *** priteau has quit IRC | 13:15 | |
| *** ijw has quit IRC | 13:15 | |
| *** owalsh has joined #openstack-infra | 13:15 | |
| *** ijw has joined #openstack-infra | 13:16 | |
| *** priteau has joined #openstack-infra | 13:17 | |
| *** lseki has joined #openstack-infra | 13:17 | |
| *** owalsh_ has joined #openstack-infra | 13:20 | |
| *** ijw has quit IRC | 13:21 | |
| clarkb | sshnaidm|ruck: I think fungi typically handles those but is on vacation this week. If you drop details here he may be able to take care of it or point the rest of us in the right direction | 13:22 |
| clarkb | I'm guessing this is similar to merging gerrit accounts by pointing extra openid at correct account? | 13:23 |
| *** owalsh has quit IRC | 13:23 | |
| *** owalsh_ is now known as owalsh | 13:23 | |
| *** ramishra has quit IRC | 13:26 | |
| *** ramishra has joined #openstack-infra | 13:28 | |
| *** lpetrut has joined #openstack-infra | 13:34 | |
| fungi | i have a moment to take a look | 13:34 |
| fungi | but yes, it's generally the same problem as we see on gerrit... launchpad sso providing a new openid for an existing user, so storyboard thinks it's a new user and autocreates an account for them | 13:35 |
| sshnaidm|ruck | fungi, clarkb the user "id: 3222, email: Sagi (Sergey) Shnaidman-dc9694e8-54f1-49ff-a820-ec0c777d43f2@example.com" is redundant there | 13:38 |
| *** bhavikdbavishi has quit IRC | 13:41 | |
| fungi | sshnaidm|ruck: yep, that account was unused so i've set it inactive | 13:44 |
| sshnaidm|ruck | fungi, thanks | 13:45 |
| *** psachin has quit IRC | 13:47 | |
| donnyd | clarkb: I have been toying around with the idea of running the mirror on an ironic instance? Any interest? | 13:47 |
| *** ijw has joined #openstack-infra | 13:48 | |
| *** zzzeek has quit IRC | 13:49 | |
| donnyd | This way I could make the important parts of the instance faster | 13:49 |
| *** zzzeek has joined #openstack-infra | 13:51 | |
| clarkb | donnyd: I dont think that is necessary. Its basically a cache that we ship into a region so doesnt need much | 13:51 |
| *** zzzeek has quit IRC | 13:53 | |
| donnyd | What does it cache? | 13:54 |
| *** zzzeek has joined #openstack-infra | 13:54 | |
| *** zzzeek has quit IRC | 13:55 | |
| clarkb | the contents of our afs filesystems (linix distro repo mirrors mostly) as well as dockerhub and pypi and other types of artifact repos | 13:55 |
| *** zzzeek has joined #openstack-infra | 13:56 | |
| donnyd | Well I was thinking we could improve build performance by making that instance's backend reside on a dedicated nvme drive... making read's much faster | 13:56 |
| clarkb | network io tends to be our bottleneck there and the VMs seems to have great network ii | 13:56 |
| *** sgw has joined #openstack-infra | 13:57 | |
| slaweq | hi infra-root, I have a question | 13:57 |
| donnyd | well we will see how good the network is when we get a workload on it | 13:57 |
| donnyd | LOL | 13:57 |
| *** Fidde has joined #openstack-infra | 13:57 | |
| clarkb | and I think I'd like to avoid making this one mirror node significa tly different than all the others we deploy | 13:57 |
| slaweq | can You take a look at https://review.opendev.org/#/c/643733/ and tell me there if in Your opinion it may be useful or maybe should I abandon it :) | 13:58 |
| slaweq | thx in advance | 13:58 |
| donnyd | Ok, just thought I would offer | 13:59 |
| donnyd | :) | 13:59 |
| *** mriedem has joined #openstack-infra | 13:59 | |
| *** rfarr__ has quit IRC | 14:00 | |
| donnyd | I have been running some build tests this morning to see what happens | 14:01 |
| *** rfolco has joined #openstack-infra | 14:01 | |
| frickler | slaweq: since it looks like you are mainly interested in getting logs for some specific services which are currently not covered, I think it might be more useful to just amend the log-collection role in devstack accordingly | 14:02 |
| *** rfarr has joined #openstack-infra | 14:02 | |
| slaweq | frickler: ok, I will do it that way, thx | 14:08 |
| *** ginopc has quit IRC | 14:10 | |
| *** ijw has quit IRC | 14:11 | |
| AJaeger | frickler, could you review https://review.opendev.org/#/c/668061, please? That's the next change in the zuul-jobs testing change series. | 14:11 |
| *** ijw has joined #openstack-infra | 14:11 | |
| *** ginopc has joined #openstack-infra | 14:13 | |
| *** ricolin has quit IRC | 14:16 | |
| *** jaosorior has quit IRC | 14:20 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 14:23 |
| AJaeger | clarkb: what do you think about adding a section to infra manual about pushing to github? After seeing that question twice during the last 24 hours... | 14:24 |
| clarkb | AJaeger: probably a good idea. I guess that would be a project creator entry? | 14:25 |
| clarkb | donnyd: openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://openstack.fortnebula.com:13696/v2.0/floatingips.json, No more IP addresses available on network 7777a6ef-2d07-49fd-b109-4223b5df3bf0 | 14:26 |
| clarkb | donnyd: possibly due to your testing all the IPs are used up? | 14:26 |
| AJaeger | clarkb: yes | 14:26 |
| donnyd | Yea | 14:26 |
| donnyd | Lol | 14:26 |
| clarkb | we also apparently fail to delete the instance we boot when we hit that error | 14:27 |
| * clarkb cleans up that instance and waits for IPs to be free | 14:27 | |
| donnyd | that project already has ip's allocated | 14:28 |
| clarkb | ok trying again then | 14:28 |
| donnyd | I will release it, as I am assuming in the instance boot it's trying to grab one | 14:28 |
| clarkb | ya its trying to grab one as part of our provisioning | 14:29 |
| donnyd | yea, i will just release it back to the pool and it will grab it on boot | 14:29 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 14:31 |
| donnyd | Ok, i released all i was using back into the pool as well | 14:32 |
| *** smarcet has joined #openstack-infra | 14:32 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base integration roles https://review.opendev.org/668061 | 14:33 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration roles https://review.opendev.org/668767 | 14:33 |
| donnyd | It looks like its good to go. I am tearing down all 100 of my test instances | 14:33 |
| clarkb | donnyd: any idea why it didn't also get an ipv6 address? | 14:34 |
| clarkb | (I think we want both so that we don't have to go through ipv4 NAT for mirror access from the test nodes, they'll prefer ipv6 if it is there) | 14:34 |
| donnyd | I can make it be there | 14:35 |
| clarkb | that would be great | 14:35 |
| clarkb | I can easily rebuild the instance too if necessary | 14:35 |
| donnyd | Its no issue | 14:35 |
| donnyd | do you want it on the same v6 network as zuul? | 14:36 |
| donnyd | or different subnet? | 14:36 |
| clarkb | I think same is fine | 14:36 |
| clarkb | we firewall all our hosts so aren't relying on external controls for that | 14:36 |
| clarkb | donnyd: should I expect these changes to be reflected in the nova listing of the instance when done? (I expect neutron will sort that out for us?) | 14:41 |
| donnyd | Yea, you should just get an ipv6 address | 14:42 |
| donnyd | I need to really get bgp running | 14:42 |
| donnyd | Its on the list | 14:42 |
| openstackgerrit | Merged zuul/zuul master: model: add cleanup-run to the job configuration https://review.opendev.org/661880 | 14:43 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Document git repo mirroring https://review.opendev.org/668927 | 14:45 |
| AJaeger | clarkb: ^ | 14:45 |
| clarkb | ok neutron knows about the address but the instance doesn't yet. I think the instance may just need to wait for an RA to happen | 14:45 |
| AJaeger | dmsimard: I added you as coauthor on 668927 | 14:45 |
| clarkb | once the instance notices it has a new ipv6 addr I'll push dns updates, then we can add it to the ansible inventory then we should have a funcioning mirror and can start configuring nodepool | 14:46 |
| clarkb | AJaeger: lgtm with one minor suggestion | 14:47 |
| clarkb | donnyd: instance knows about its IPv6 address now. But I cannot ssh to it yet. traceroute indicates we get lost on an he tunnel somewhere | 14:48 |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files. Added MYSQL dependencies in order to run unit IDP unit test. Added php unit invocations for main tests * OIDC * OAUTH2 * OPENID https://review.opendev.org/668772 | 14:48 |
| clarkb | perhaps fallout from the power outage? | 14:48 |
| donnyd | yea, I am still working on it | 14:48 |
| donnyd | This is why I need to get bgp setup... | 14:49 |
| donnyd | So when I provision a v6 network its done | 14:49 |
| donnyd | just haven't gotten to it yet | 14:49 |
| clarkb | ah | 14:49 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add fortnebula mirror server to DNS https://review.opendev.org/668929 | 14:50 |
| *** igordc has joined #openstack-infra | 14:50 | |
| *** bnemec has quit IRC | 14:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Document git repo mirroring https://review.opendev.org/668927 | 14:51 |
| AJaeger | clarkb: thanks for review and suggestion, I addressed it ^ | 14:51 |
| openstackgerrit | Merged zuul/zuul master: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 14:51 |
| *** factor has quit IRC | 14:51 | |
| *** factor has joined #openstack-infra | 14:52 | |
| *** jpich has quit IRC | 14:54 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add fortnebula ci mirror https://review.opendev.org/668931 | 14:54 |
| clarkb | infra-root ^ and its parents should all be ready to go now | 14:54 |
| *** jpich has joined #openstack-infra | 14:54 | |
| clarkb | that will get us a working mirror and configured cloud (security groups and ssh keys) then I can get some nodepool config up | 14:54 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Document git repo mirroring https://review.opendev.org/668927 | 14:55 |
| AJaeger | clarkb, corvus, adressed corvus suggestion ^ | 14:56 |
| clarkb | +2 | 14:56 |
| AJaeger | thanks, clarkb | 14:56 |
| clarkb | now for some tea | 14:58 |
| AJaeger | Enjoy! | 14:58 |
| *** icarusfactor has joined #openstack-infra | 15:00 | |
| *** factor has quit IRC | 15:00 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 15:02 |
| *** bobh has joined #openstack-infra | 15:05 | |
| clarkb | mordred: can you look at https://review.opendev.org/#/c/668251/2 too? its the bottom of that fortnebula stack and needs approval before the others | 15:06 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add fortnebula mirror server to DNS https://review.opendev.org/668929 | 15:06 |
| *** mattw4 has joined #openstack-infra | 15:06 | |
| *** mattw4 has quit IRC | 15:09 | |
| donnyd | clarkb: Can you ifup / down or reboot the instance so the v6 address populates | 15:09 |
| donnyd | should be good now | 15:09 |
| *** mattw4 has joined #openstack-infra | 15:09 | |
| *** mattw4 has quit IRC | 15:11 | |
| clarkb | rebooted and interface is configured. Not able to ssh to it from my ovh vm yet though. I wonder if that is a security group problem? if it is our change linked above will correct that | 15:12 |
| * clarkb looks at security groups rules | 15:12 | |
| AJaeger | infra-root, https://opendev.org/openstack/project-config/ is outdated, it misses a couple of changes. Looking at https://review.opendev.org/#/q/project:openstack/project-config+is:merged, the last three changes are not in opendev for me | 15:12 |
| *** mattw4 has joined #openstack-infra | 15:12 | |
| *** Lucas_Gray has quit IRC | 15:13 | |
| *** Wryhder has joined #openstack-infra | 15:13 | |
| donnyd | try again | 15:13 |
| *** priteau has quit IRC | 15:14 | |
| *** Wryhder is now known as Lucas_Gray | 15:14 | |
| clarkb | AJaeger: we seem to haev a backlog of gerrit git pushes for replication starting at 08:10:40.378 | 15:14 |
| AJaeger | thats long ago - anything stuck? | 15:14 |
| clarkb | this includes git push to github | 15:15 |
| clarkb | (which implies that maybe gitea isn't at fault here) | 15:15 |
| clarkb | looks like tricircle is the repo it first got stuck on | 15:15 |
| clarkb | donnyd: works now thanks | 15:15 |
| clarkb | AJaeger: let me look at the replication log | 15:15 |
| AJaeger | thanks, clarkb | 15:16 |
| openstackgerrit | Merged zuul/zuul master: docs: add cleanup-run documentation https://review.opendev.org/662147 | 15:16 |
| openstackgerrit | Merged openstack/infra-manual master: Document git repo mirroring https://review.opendev.org/668927 | 15:18 |
| AJaeger | clarkb: wow, 1200+ entries in the queue ;( | 15:19 |
| clarkb | If I'm reading the log right its actually still trying to replicate tricircle | 15:19 |
| clarkb | for the last 7 hours. Not failing then retrying but still on the initial push | 15:20 |
| clarkb | I think we may want to stop those tasks and see if the queue flushes and if it does manually ask gerrit to replicate tricircle again | 15:20 |
| clarkb | infra-root^ any other ideas or objections to that plan? | 15:20 |
| AJaeger | clarkb: there are also four entries from early June - are those problems as well? | 15:20 |
| clarkb | AJaeger: the git-upload-packs? I don't think so as those are for people fetching refs | 15:21 |
| *** jamesmcarthur has joined #openstack-infra | 15:21 | |
| clarkb | we can probably just kill those too since its been days | 15:21 |
| *** jamesmcarthur has quit IRC | 15:21 | |
| clarkb | I'm guessing they had clients not cleanly close the requests and gerrit is being helpful by waiting | 15:21 |
| *** jamesmcarthur has joined #openstack-infra | 15:22 | |
| *** liuyulong has joined #openstack-infra | 15:22 | |
| AJaeger | clarkb: ah, I see. Yes, we can kill those... | 15:24 |
| corvus | clarkb: looking | 15:25 |
| corvus | clarkb: there are a bunch of parallel replication tasks that are stuck... it may be possible they're all waiting on a mutex and killing that thread may not release it | 15:26 |
| corvus | clarkb: i don't have any better ideas, but i think we should be prepared for a restart if killing tasks doesn't unbreak the jam | 15:27 |
| clarkb | corvus: ok I'll start by killing the queued tasks | 15:27 |
| clarkb | (and a restart will do a full replication so if we end up there we should be fine after) | 15:27 |
| *** priteau has joined #openstack-infra | 15:28 | |
| *** bnemec has joined #openstack-infra | 15:29 | |
| *** jcoufal_ has joined #openstack-infra | 15:29 | |
| *** liuyulong has quit IRC | 15:30 | |
| *** gyee has joined #openstack-infra | 15:31 | |
| *** jcoufal has quit IRC | 15:32 | |
| *** icarusfactor has quit IRC | 15:34 | |
| *** factor__ has joined #openstack-infra | 15:34 | |
| AJaeger | clarkb: seems to be moving forward, isn't it? | 15:34 |
| clarkb | ok I claered all the stuck tasks | 15:34 |
| clarkb | I took notes of which projects those tasks were cleared for so they can be replicated once things settle | 15:34 |
| clarkb | AJaeger: all of the git pushes still show "waiting" | 15:35 |
| clarkb | so I think corvus' concern may be our reality | 15:36 |
| *** lmiccini has quit IRC | 15:36 | |
| clarkb | one other thing we can attempt to do is update the replication config so that it notices and maybe that will unlock whatever lock is blocking us? | 15:36 |
| clarkb | though at that point restarting the server may be easier | 15:36 |
| AJaeger | clarkb: it's moving forward, isn't it - a "gerrit show-queue|head" shows that at least | 15:37 |
| AJaeger | clarkb: but I'm not an expert, so might misread the output | 15:38 |
| clarkb | AJaeger: 07f7c8f9 that task at the head of the queue is still "waiting though" | 15:38 |
| clarkb | er "waiting" | 15:38 |
| clarkb | so the next push tasks aren't being assigned to workers | 15:38 |
| *** factor has joined #openstack-infra | 15:38 | |
| *** factor__ has quit IRC | 15:39 | |
| AJaeger | clarkb: I misread - the lines change are ssh replications by users ;( | 15:39 |
| AJaeger | clarkb: so, agreed, time to restart ;( | 15:39 |
| clarkb | we can wait for those two changes to merge so that they don't fail to merge while gerrit is down | 15:39 |
| clarkb | but then ya we should restart the service | 15:39 |
| clarkb | infra-root ^ fyi | 15:39 |
| *** udesale has quit IRC | 15:45 | |
| *** rfolco has quit IRC | 15:47 | |
| *** jpich has quit IRC | 15:48 | |
| *** jpich has joined #openstack-infra | 15:48 | |
| clarkb | unfortunately that 0 minutes remaining has become more than a few minutes :/ thats ok I'll be patient | 15:50 |
| *** apetrich has quit IRC | 15:55 | |
| clarkb | ok tripleo changes merged | 15:55 |
| clarkb | I'm going to stop gerrit then start it and it should reenqueue replication on its own | 15:55 |
| clarkb | here goes | 15:55 |
| *** pramchan89 has joined #openstack-infra | 15:56 | |
| *** bobh has quit IRC | 15:58 | |
| clarkb | #status log Restarted review.opendev.org's Gerrit service to restore git repo replication which appears to have deadlocked. | 15:58 |
| openstackstatus | clarkb: finished logging | 15:58 |
| clarkb | the task count while now larger is trending in the correct direction | 15:59 |
| *** pgaxatte has quit IRC | 16:02 | |
| *** Lucas_Gray has quit IRC | 16:02 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Upload test images to fortnebula cloud https://review.opendev.org/668952 | 16:03 |
| clarkb | infra-root ^ another change in bootstrapping fortnebula. I think that is the last one I can push up before we have the mirror running (which needs https://review.opendev.org/#/c/668251/2) | 16:04 |
| clarkb | also I'll be adding this cloud to nl02 which is largely idle now that limestone has been pulled from rotation | 16:04 |
| clarkb | donnyd: ^ fyi slow by steady progress. I figured we'd start with ~10 test nodes, double check jobs look happy then ramp that up to whatever number you are able to support? I think you mentioned 100 earlier? | 16:05 |
| donnyd | I think that should be ok, but while I am out this weekend I would probably keep the number a little lower | 16:05 |
| *** Fidde has quit IRC | 16:06 | |
| clarkb | donnyd: ya I'll be out too so no problem with that | 16:06 |
| *** apetrich has joined #openstack-infra | 16:06 | |
| *** lucasagomes has quit IRC | 16:07 | |
| donnyd | Is the mirror good to go? | 16:07 |
| clarkb | donnyd: no waiting on parent chagnes to get in so that it will get added to the inventory | 16:08 |
| clarkb | https://review.opendev.org/#/c/668251/2 specifically | 16:08 |
| clarkb | once added to thei nventory our ansible will run and provision ssl certs and configure the web server | 16:08 |
| corvus | clarkb: well, it seems like we're the only ones around so i will +1 my +2 | 16:08 |
| clarkb | corvus: ok thanks | 16:08 |
| *** hemna has quit IRC | 16:08 | |
| clarkb | I had seen mordred aroudn but awsn't sure if he was paying attention | 16:09 |
| * mordred is in the midst of lunching ... looking though just to be sure | 16:09 | |
| mordred | +A | 16:09 |
| corvus | cool, looks like we're all caught up and can +W the mirror after dns happens | 16:10 |
| clarkb | ++ | 16:10 |
| clarkb | oh that may actually be stuck behind replication too :/ | 16:10 |
| clarkb | dns might be one of those things we want to fetch repos directly from gerrit for | 16:11 |
| clarkb | down to 1600 tasks from 17881 | 16:11 |
| clarkb | so progress on that front as well and I shall practice patience | 16:11 |
| *** smarcet has quit IRC | 16:12 | |
| *** smarcet has joined #openstack-infra | 16:13 | |
| *** rpittau is now known as rpittau|afk | 16:14 | |
| cloudnull | Hey all, we just started running into the following error with our releasenotes jobs http://logs.openstack.org/63/668063/17/check/build-reno-releasenotes/8177c48/job-output.txt.gz#_2019-07-03_14_21_15_581879 is that something known at this point? | 16:14 |
| cloudnull | seems to be failing in opendev.org/zuul/zuul-jobs/playbooks/releasenotes/pre.yaml | 16:15 |
| clarkb | cloudnull: need to pin sphinx to <2.0 when using python2 | 16:16 |
| clarkb | its a known issue, but I think new to this job | 16:16 |
| clarkb | I believe most projects are using python3 for that job | 16:16 |
| *** tdasilva has quit IRC | 16:17 | |
| cloudnull | we're just including the job in our layout, should we be augmenting it with some var? | 16:17 |
| clarkb | cloudnull: sphinx_python is the var | 16:18 |
| clarkb | release-notes-jobs-python3 is the template most projects are using I think | 16:19 |
| cloudnull | oic | 16:19 |
| cloudnull | we just have build-reno-releasenotes | 16:19 |
| clarkb | my initial thought was we can probably use env markers in the package list to specify different sphinx versions for different pythons but reading the role it uses the ansible pip module to installthem and if the env marker doesn't match I expect that will fail | 16:19 |
| clarkb | no longer sure about the best way to fix that for python2 | 16:20 |
| clarkb | maybe a switch to set different list values depending on the value of sphinx_python | 16:20 |
| clarkb | if starts with python or starts with python2: package = sphinx<2.0 else sphinx | 16:21 |
| clarkb | 14.5k replication tasks now | 16:21 |
| *** apetrich has quit IRC | 16:22 | |
| clarkb | and now under 14k. Since that hasn't gone sideways yet I'll stop watching it and take this as an opportunity to get a bike ride in. Will be back to get the mirror ansibled and hopefully nodepool nodepooling | 16:26 |
| openstackgerrit | Merged openstack/project-config master: Upload test images to fortnebula cloud https://review.opendev.org/668952 | 16:27 |
| *** priteau has quit IRC | 16:28 | |
| *** jpich has quit IRC | 16:32 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add python3 detection to the sphinx role https://review.opendev.org/668954 | 16:33 |
| *** bobh has joined #openstack-infra | 16:33 | |
| cloudnull | clarkb ^ | 16:34 |
| cloudnull | actually the sphinx version change would be a good thing to have too. | 16:34 |
| * cloudnull updating | 16:34 | |
| openstackgerrit | Merged opendev/system-config master: Add fortnebula cloud to cloud launcher https://review.opendev.org/668251 | 16:35 |
| pabelanger | cloudnull: mordred: soft -1 on ^ | 16:38 |
| *** Fidde has joined #openstack-infra | 16:38 | |
| pabelanger | I think we had an issue using command in a zuul-job, because it is a bashism | 16:38 |
| cloudnull | I could use which ? | 16:38 |
| *** bobh has quit IRC | 16:38 | |
| clarkb | pabelanger: yes we did. command is a sehll built in so should be run in shell task not command task | 16:38 |
| clarkb | also it isn't available in normal bourne but is in dash and bash so for many systems will work fine in that case | 16:39 |
| *** e0ne has quit IRC | 16:39 | |
| clarkb | cloudnull: ^ | 16:39 |
| clarkb | ok really popping out now. back soon | 16:39 |
| *** rkukura has quit IRC | 16:40 | |
| openstackgerrit | Merged opendev/system-config master: Use qcow2 images on fortnebula https://review.opendev.org/668757 | 16:41 |
| *** rh-jelabarre has quit IRC | 16:43 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add python3 detection to the sphinx role https://review.opendev.org/668954 | 16:43 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add python3 detection to the sphinx role https://review.opendev.org/668954 | 16:44 |
| cloudnull | pabelanger ^ that should work in much the same way | 16:44 |
| *** bnemec has quit IRC | 16:45 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base integration roles https://review.opendev.org/668061 | 16:47 |
| *** ginopc has quit IRC | 16:47 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration job https://review.opendev.org/668767 | 16:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add a script to make platform-specific versions of jobs https://review.opendev.org/668955 | 16:47 |
| *** eernst has joined #openstack-infra | 16:48 | |
| smarcet | clarkb: fungi: hello guys hope that everything is ok on your side, i merged this change https://review.opendev.org/#/c/668926/ | 16:48 |
| smarcet | but is not replicating to osf repo here https://opendev.org/osf/openstackid-resources/commits/branch/master | 16:48 |
| *** rh-jelabarre has joined #openstack-infra | 16:49 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base role integration jobs https://review.opendev.org/668061 | 16:52 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 16:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration jobs https://review.opendev.org/668767 | 16:52 |
| *** rkukura has joined #openstack-infra | 16:54 | |
| *** jamesmcarthur has quit IRC | 16:56 | |
| *** jamesmcarthur has joined #openstack-infra | 16:57 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files. https://review.opendev.org/668772 | 16:57 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add python3 detection to the sphinx role https://review.opendev.org/668954 | 16:59 |
| cloudnull | ^ Updated to ensure things in el7 w/ software factory are in a good state. | 17:00 |
| *** jamesmcarthur has quit IRC | 17:02 | |
| *** markvoelker has joined #openstack-infra | 17:02 | |
| *** factor has quit IRC | 17:03 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add test job for ensure-sphinx https://review.opendev.org/668958 | 17:03 |
| *** kjackal has quit IRC | 17:04 | |
| *** openstackgerrit has quit IRC | 17:04 | |
| *** witek has quit IRC | 17:04 | |
| corvus | cloudnull: ^ that's some *very* basic testing of that role -- i have some other changes in flight that make it easy to test that it on all our platforms ... maybe that would be useful too? | 17:04 |
| cloudnull | ++ | 17:05 |
| corvus | cloudnull: (we're currently reinventing how to do testing in that repo) | 17:05 |
| *** elod has quit IRC | 17:05 | |
| cloudnull | ah, well if I can help/review let me know | 17:06 |
| corvus | will do | 17:06 |
| *** elod has joined #openstack-infra | 17:07 | |
| yoctozepto | any idea why Zuul did not pick up this: https://review.opendev.org/667971 for gating? I ran recheck now and hope it gets to gating then ;/ | 17:07 |
| *** openstackgerrit has joined #openstack-infra | 17:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add a script to make platform-specific versions of jobs https://review.opendev.org/668955 | 17:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base role integration jobs https://review.opendev.org/668061 | 17:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration jobs https://review.opendev.org/668767 | 17:13 |
| *** rfarr_ has joined #openstack-infra | 17:14 | |
| *** bnemec has joined #openstack-infra | 17:14 | |
| *** lpetrut has quit IRC | 17:16 | |
| *** rfarr has quit IRC | 17:17 | |
| corvus | yoctozepto: the depends-on change had not merged at the time 667971 was approved; consider putting kolla-ansible and kolla in the same zuul change queue so that it can gate those changes together | 17:17 |
| *** rfarr__ has joined #openstack-infra | 17:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add test job for ensure-sphinx https://review.opendev.org/668958 | 17:20 |
| *** rfarr_ has quit IRC | 17:21 | |
| *** ykarel is now known as ykarel|away | 17:23 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Removed environment.php file due its been a while that LV fwk moved to .env files. https://review.opendev.org/668772 | 17:25 |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Update legacy laravel unit test Job for LV 5.x https://review.opendev.org/668772 | 17:26 |
| *** ociuhandu has quit IRC | 17:26 | |
| openstackgerrit | sebastian marcet proposed openstack/openstack-zuul-jobs master: Update legacy laravel unit test Job for LV 5.x https://review.opendev.org/668772 | 17:27 |
| yoctozepto | > <corvus> yoctozepto: the depends-on change had not merged at the time 667971 was approved; consider putting kolla-ansible and kolla in the same zuul change queue so that it can gate those changes together | 17:27 |
| yoctozepto | how do I do that? :O | 17:28 |
| *** jamesmcarthur has joined #openstack-infra | 17:28 | |
| corvus | yoctozepto: https://docs.openstack.org/infra/manual/creators.html#shared-queues-for-cross-project-testing | 17:34 |
| clarkb | smarcet: tgere is a replication backlog | 17:35 |
| *** ramishra has quit IRC | 17:37 | |
| clarkb | down to 3k of 17.8k tasks | 17:37 |
| *** weifan has joined #openstack-infra | 17:37 | |
| clarkb | dns appears to have updated. I have approved https://review.opendev.org/#/c/668931/ | 17:38 |
| clarkb | hrm there is a race between updating nodepools config and clouds.yaml | 17:41 |
| yoctozepto | corvus: thanks! | 17:41 |
| clarkb | that means initial images being uplaoded to fortnebula are raw and now qcow2 | 17:41 |
| clarkb | I don't think this is amajor issue, it will flip over starting tomorrow | 17:41 |
| donnyd | https://usercontent.irccloud-cdn.com/file/Na1eFWux/image.png | 17:43 |
| donnyd | Yea, they are all being loaded now | 17:43 |
| *** ykarel|away has quit IRC | 17:43 | |
| clarkb | donnyd: and change to configure the mirror is on its way in | 17:43 |
| yoctozepto | corvus: queue name is arbitrary I presume? | 17:43 |
| donnyd | Now things are starting to get exciting | 17:44 |
| donnyd | :) | 17:44 |
| *** e0ne has joined #openstack-infra | 17:44 | |
| *** jamesmcarthur has quit IRC | 17:44 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base role integration jobs https://review.opendev.org/668061 | 17:44 |
| corvus | yoctozepto: yes | 17:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration jobs https://review.opendev.org/668767 | 17:44 |
| yoctozepto | corvus: thanks! | 17:44 |
| *** igordc has quit IRC | 17:45 | |
| clarkb | cloudnull: that proposed change is not backward compatible | 17:45 |
| *** ijw_ has joined #openstack-infra | 17:45 | |
| clarkb | cloudnull: any jobs that previously relied on the default of python2 will get python3 almost everywhere | 17:46 |
| clarkb | and I don't think it solves your problem? | 17:46 |
| clarkb | any jobs running on python2 will still install the bad version of sphinx | 17:46 |
| clarkb | (well the existing jobs that specify packages) | 17:46 |
| donnyd | Going to grab lunch real quick. Will be back | 17:48 |
| *** ijw has quit IRC | 17:49 | |
| cloudnull | clarkb any jobs running on py2 should continue to do so? | 17:52 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 17:53 | |
| cloudnull | it'd now just look for py3 and if found use that instead, with a known good combination of sphinx | 17:53 |
| cloudnull | unless the sphinx_python is defined, then it'd use whatever was set | 17:54 |
| clarkb | cloudnull: yes the old default was python2 so yuo didn't have to define it | 17:54 |
| *** rfolco has joined #openstack-infra | 17:56 | |
| *** weifan has quit IRC | 17:57 | |
| cloudnull | I guess I could make it run py2 by default? | 17:57 |
| *** weifan has joined #openstack-infra | 17:57 | |
| *** weifan has quit IRC | 17:58 | |
| *** weifan has joined #openstack-infra | 17:58 | |
| clarkb | It already does that. But I also don't knwo that this solves the problem we have | 17:58 |
| clarkb | because its the consumer of the role providing the python version and the deps to install | 17:58 |
| cloudnull | that change solves my specific problem, an incompatible sphinx + py combination | 17:58 |
| clarkb | cloudnull: only because it defaults you to python3 | 17:58 |
| *** weifan has quit IRC | 17:58 | |
| clarkb | but doesn't fix it for anyone on python2 | 17:58 |
| *** weifan has joined #openstack-infra | 17:58 | |
| clarkb | (because they are still providing their own package lists with bad sphinx) | 17:59 |
| cloudnull | https://review.opendev.org/#/c/668954/4/roles/ensure-sphinx/vars/python2.yaml | 17:59 |
| cloudnull | that should do the trick when py2 is defined without a package list | 17:59 |
| clarkb | ya but that doesn't help when the jobs are still providing bad dep list | 17:59 |
| *** weifan has quit IRC | 17:59 | |
| clarkb | which is how the jobs are set up | 17:59 |
| *** weifan has joined #openstack-infra | 17:59 | |
| clarkb | I think we want to update the thing calling the role to specify valid deps | 17:59 |
| *** weifan has quit IRC | 18:00 | |
| *** weifan has joined #openstack-infra | 18:00 | |
| cloudnull | oh. I think if folks feed in bad deps im not sure we should do much to correct it, beside have the job fail. maybe in a more graceful way. | 18:01 |
| *** weifan has quit IRC | 18:01 | |
| clarkb | yup exactly. The role works fine as is | 18:01 |
| *** weifan has joined #openstack-infra | 18:01 | |
| clarkb | we should just update where we use the role to feed in valid deps | 18:01 |
| *** weifan has quit IRC | 18:01 | |
| *** weifan has joined #openstack-infra | 18:02 | |
| cloudnull | it doesn't work right now. if sphinx_python is undefined it defaults to py2 and trys to install an incompatible sphinx | 18:02 |
| *** weifan has quit IRC | 18:02 | |
| clarkb | right but its failing correctly | 18:02 |
| *** weifan has joined #openstack-infra | 18:02 | |
| clarkb | so fix is just updte dep list | 18:02 |
| cloudnull | so as a user I have to define sphinx_python or doc_building_packages I can't accept the defaults | 18:03 |
| *** weifan has quit IRC | 18:03 | |
| *** weifan has joined #openstack-infra | 18:03 | |
| cloudnull | so your saying we should just pin doc_building_packages for py2? | 18:03 |
| clarkb | but those aren't defaults we are defining them https://opendev.org/zuul/zuul-jobs/src/branch/master/playbooks/releasenotes/pre.yaml#L7-L9 | 18:04 |
| clarkb | I'm saying ^ is what needs to update | 18:04 |
| clarkb | otherwise your job will continue to fail | 18:04 |
| *** weifan has quit IRC | 18:04 | |
| clarkb | (if you specify python2, or possibly fail because your code doesn't build with python3) | 18:04 |
| *** weifan has joined #openstack-infra | 18:04 | |
| *** weifan has quit IRC | 18:05 | |
| cloudnull | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-sphinx/defaults/main.yaml#L5 | 18:05 |
| *** weifan has joined #openstack-infra | 18:05 | |
| cloudnull | that's the bad default when using py2 | 18:05 |
| clarkb | ah ok so we are broken in both places | 18:05 |
| cloudnull | that would need to be sphinx<2 | 18:05 |
| clarkb | in that case I would change doc_building_packages in your link to default to sphinx<2.0 since we also default to python2 | 18:05 |
| cloudnull | ++ | 18:05 |
| cloudnull | will do | 18:05 |
| *** weifan has quit IRC | 18:05 | |
| *** e0ne has quit IRC | 18:06 | |
| clarkb | then the defaults will work and we don't potentially break anyone whos code would fail under python3 | 18:06 |
| *** e0ne has joined #openstack-infra | 18:09 | |
| *** weifan has joined #openstack-infra | 18:09 | |
| AJaeger | clarkb: down to one gerrit replication task - yeah! | 18:14 |
| openstackgerrit | Merged opendev/system-config master: Add fortnebula ci mirror https://review.opendev.org/668931 | 18:14 |
| clarkb | AJaeger: \o/ | 18:15 |
| AJaeger | clarkb: we're through, now we have the normal ones ;) | 18:15 |
| *** ijw_ has quit IRC | 18:20 | |
| *** mattw4 has quit IRC | 18:21 | |
| *** ijw has joined #openstack-infra | 18:21 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base role integration jobs https://review.opendev.org/668061 | 18:21 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add multi-node integration jobs https://review.opendev.org/668767 | 18:21 |
| *** e0ne has quit IRC | 18:22 | |
| *** ijw has quit IRC | 18:24 | |
| *** ijw has joined #openstack-infra | 18:24 | |
| *** rascasoft has quit IRC | 18:25 | |
| *** rascasoft has joined #openstack-infra | 18:25 | |
| *** kjackal has joined #openstack-infra | 18:27 | |
| *** _erlon_ has quit IRC | 18:28 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Update the sphinx python package sets https://review.opendev.org/668954 | 18:28 |
| *** hwoarang has quit IRC | 18:31 | |
| *** weifan has quit IRC | 18:33 | |
| *** raissa has joined #openstack-infra | 18:33 | |
| *** hwoarang has joined #openstack-infra | 18:34 | |
| *** pramchan89 has quit IRC | 18:34 | |
| *** ralonsoh has quit IRC | 18:35 | |
| *** kjackal has quit IRC | 18:36 | |
| *** weifan has joined #openstack-infra | 18:38 | |
| *** raissa has quit IRC | 18:42 | |
| *** tesseract has quit IRC | 18:44 | |
| *** weifan has quit IRC | 18:49 | |
| *** mattw4 has joined #openstack-infra | 18:54 | |
| smarcet | clarkb: i got this error out of the blue, http://logs.openstack.org/61/661661/23/check/tox-docs/2b776a1/tox/docs-1.log its complaning about that sphinx its expecting python 3.5, but its getting 2.7 | 18:56 |
| *** iurygregory has joined #openstack-infra | 18:56 | |
| smarcet | clarkb: i guess that requirement.txt is too generic ? should i tied to sphinx=1.1.2? | 18:57 |
| clarkb | smarcet: for python2 you have to specify sphinx<2.0 | 18:58 |
| smarcet | oh ok | 18:58 |
| smarcet | thx for the finp | 18:58 |
| smarcet | info | 18:59 |
| clarkb | ansible should run on our new mirror shortly (it just started getting base applied to it) | 19:03 |
| donnyd | clarkb: where can you see the status of such things | 19:05 |
| *** igordc has joined #openstack-infra | 19:05 | |
| clarkb | donnyd: unfortunately we don't have that exposed currently | 19:05 |
| clarkb | donnyd: our ansible tooling runs in a cron and I tailed its logs :/ | 19:05 |
| clarkb | Jul 3 19:10:27 mirror01 python3[1944]: ansible-apt [WARNING] Could not find aptitude. Using apt-get instead | 19:10 |
| clarkb | anyone know why that is a warning? | 19:10 |
| clarkb | like is apt-get less capable with the ansible module or something? | 19:11 |
| *** whoami-rajat has quit IRC | 19:12 | |
| donnyd | https://github.com/ansible/ansible/issues/56832 | 19:12 |
| donnyd | looks like its just warns to warn | 19:12 |
| clarkb | https://github.com/ansible/ansible/issues/18987 apparently that is the underlying issue that has them recommend aptitude? | 19:14 |
| clarkb | might be the sort of thing to talk to people at ansiblefest about :/ | 19:14 |
| clarkb | since all the issues are apparently still present but closed and the fix is to warn peopel to use aptitude | 19:15 |
| clarkb | which if that is the most reliable method then maybe we should use it, but aptitude isn't included by default on debuntu as far as I know | 19:15 |
| mordred | it seems weird to me that apt-get would be considered unreliable | 19:19 |
| *** markvoelker has quit IRC | 19:19 | |
| clarkb | mordred: the issue seems to be that apt-get has the update race and/or problems | 19:19 |
| clarkb | whereas aptitude knows how to do that more atomically supposedly | 19:19 |
| mordred | I've never switched to aptitude because it's more letters to type and it always feels weird for some reason | 19:19 |
| *** markvoelker has joined #openstack-infra | 19:19 | |
| *** smarcet has quit IRC | 19:20 | |
| mordred | "apt-get update && apt-get install foo" is pretty good at its job - but whatever | 19:20 |
| clarkb | the only thing I use aptitude for is searching | 19:21 |
| *** weifan has joined #openstack-infra | 19:22 | |
| clarkb | though supposedly its dep solver is better than apt-get's? | 19:22 |
| clarkb | like faster not necessarily more accurate | 19:22 |
| *** efried has joined #openstack-infra | 19:25 | |
| *** efried has left #openstack-infra | 19:25 | |
| *** weifan has quit IRC | 19:26 | |
| *** eharney has quit IRC | 19:27 | |
| *** iurygregory has quit IRC | 19:28 | |
| donnyd | clarkb: will the mirror job setup the security group rules entirely? | 19:29 |
| *** whoami-rajat has joined #openstack-infra | 19:29 | |
| donnyd | or do I need to add things like http or https | 19:30 |
| clarkb | donnyd: yes security groups should all be auto managed for us | 19:30 |
| clarkb | I should check that that has happened I guess | 19:30 |
| *** mattw4 has quit IRC | 19:30 | |
| *** igordc has quit IRC | 19:31 | |
| donnyd | nothing yet that I can see | 19:31 |
| gouthamr | ianw: hello o/ can you please help merge https://review.opendev.org/#/c/668787/ ---> I want to take this back to stable/queens; have some backports on manila waiting on this fix :) | 19:32 |
| clarkb | donnyd: ya hasn't updated our keys either. I'll look at thos elogs | 19:32 |
| clarkb | donnyd: I think it may not have updated because our repos were stuck behind the replication queue? | 19:36 |
| clarkb | the cron for that runs hourly and repos are up to date now so we should see it update in 24 minutes | 19:36 |
| clarkb | if it doesn't then I'll have to do more in depth debugging | 19:36 |
| *** ijw has quit IRC | 19:37 | |
| *** ijw has joined #openstack-infra | 19:37 | |
| donnyd | Sounds good | 19:38 |
| *** e0ne has joined #openstack-infra | 19:39 | |
| donnyd | I will be around if you need anything from me | 19:39 |
| *** mattw4 has joined #openstack-infra | 19:41 | |
| *** ijw has quit IRC | 19:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base role integration jobs https://review.opendev.org/668061 | 19:45 |
| clarkb | donnyd: one thing I notice is the mirror node has high cpu wai which I think is leading to higherthan expected load average for not doing much | 19:46 |
| clarkb | huh and now the wai is gone | 19:50 |
| clarkb | and ya load avg is dropping | 19:50 |
| clarkb | and then its back again weird | 19:50 |
| clarkb | fatal: [mirror01.regionone.fortnebula.opendev.org]: FAILED! => {"cache_update_time": 1562182100, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'krb5-user' 'kstart'' failed: E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)\nE: Unable to | 19:53 |
| clarkb | lock directory /var/cache/apt/archives/\n" | 19:53 |
| clarkb | could that be related? | 19:53 |
| donnyd | Not sure... I don't have access to the instance | 19:53 |
| *** dciabrin has quit IRC | 19:53 | |
| donnyd | But its a good place to start | 19:54 |
| clarkb | load average has fallen all the way back under 0.2 | 19:54 |
| donnyd | Its also doing weird things on the host... like using 5% cpu for doing nothing | 19:54 |
| donnyd | every other instance on there is at .5 maybe... | 19:54 |
| donnyd | but not this little fella | 19:54 |
| clarkb | huh | 19:55 |
| *** e0ne has quit IRC | 19:55 | |
| donnyd | should we bounce it? The bionic image that was provision is straight from upstream FYI | 19:55 |
| clarkb | http://paste.openstack.org/show/753841/ from dmesg | 19:56 |
| clarkb | I can try rebooting it | 19:56 |
| clarkb | I'll wait for you to confirm you don't need ti running for checking things on the hypervisor side | 19:57 |
| *** mattw4 has quit IRC | 19:57 | |
| donnyd | your good | 19:57 |
| *** mattw4 has joined #openstack-infra | 19:57 | |
| donnyd | I am about to be mobile | 19:57 |
| clarkb | reading the stack trace it appeas to be in ext4's sync | 19:58 |
| clarkb | so maybe the root disk for that isn't responding quickly enough | 19:58 |
| clarkb | it isn't really running any workload yet either because ansible failed installing deps for that above (it will retry again in a bit so eventually we should get ther eand that might make it easier to debug as the system will be active) | 19:58 |
| donnyd | just checking the hypervisor to make sure the mount is good and there isn't a storage issue | 20:00 |
| *** jcoufal_ has quit IRC | 20:00 | |
| clarkb | ext4 is moutned with defaults | 20:01 |
| clarkb | does that set noatime? | 20:01 |
| clarkb | but tweaking things like that might be the next step | 20:02 |
| donnyd | read: IOPS=21.7k, BW=84.8MiB/s (88.9MB/s)(1882MiB/22196msec) | 20:02 |
| donnyd | seems like the storage is working fine | 20:03 |
| clarkb | ya that should be plenty ofr journlad | 20:03 |
| clarkb | (wow I can't type) | 20:03 |
| clarkb | well we'll keep an eye on it and dig in if it persists | 20:03 |
| donnyd | ok cool | 20:03 |
| donnyd | going mobile | 20:03 |
| *** eharney has joined #openstack-infra | 20:04 | |
| clarkb | fatal: [localhost]: FAILED! => {"changed": false, "msg": "Key name bridge-root-2014-09-15 present but key hash not the same as offered. Delete key first."} is the cloud launcher issue it bails after that | 20:09 |
| clarkb | I'm going to take the suggested action and delete that key | 20:09 |
| clarkb | we may have to reorder the contents of that file and clean it up too as some of those clouds/regions don't exist anymore :/ | 20:11 |
| clarkb | I'm going to wait for it to run and confirm that for me though | 20:12 |
| clarkb | ianw: ^ when your day starts that might interest you and you may help me clean up that cloud launcher config as you've been invovled with the arm stuff | 20:14 |
| * clarkb finds lunch | 20:14 | |
| *** kjackal has joined #openstack-infra | 20:15 | |
| *** e0ne has joined #openstack-infra | 20:20 | |
| *** goldyfruit has quit IRC | 20:24 | |
| *** goldyfruit has joined #openstack-infra | 20:24 | |
| *** smarcet has joined #openstack-infra | 20:34 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 20:40 |
| *** Lucas_Gray has joined #openstack-infra | 20:42 | |
| *** trident has quit IRC | 20:43 | |
| *** weifan has joined #openstack-infra | 20:45 | |
| *** trident has joined #openstack-infra | 20:45 | |
| cloudnull | on https://review.opendev.org/#/c/668954 - zuul doesn't seem to be running any jobs and isn't reporting any output. something going on? | 20:46 |
| cloudnull | also corvus https://review.opendev.org/#/c/668958 needs a rebase, the underlying patch was changed. | 20:46 |
| cloudnull | I can do that if you dont mind ? | 20:47 |
| *** xek has quit IRC | 20:47 | |
| *** goldyfruit has quit IRC | 20:47 | |
| fungi | cloudnull: have you set any depends-on with that change or its parent which could have dependency loops? | 20:49 |
| *** weifan has quit IRC | 20:49 | |
| cloudnull | no depends-on | 20:50 |
| cloudnull | or loops that im aware of | 20:50 |
| fungi | er, child change i guess, not parent | 20:50 |
| fungi | looking in scheduler debug logs real quick | 20:51 |
| cloudnull | 668954 is the parent of 668958 | 20:51 |
| *** tdasilva has joined #openstack-infra | 20:51 | |
| fungi | right | 20:52 |
| cloudnull | but that's about it | 20:52 |
| clarkb | could be fallout from corvus' changes to how jobs on that repo are run | 20:52 |
| *** pkopec has quit IRC | 20:53 | |
| *** rh-jelabarre has quit IRC | 20:55 | |
| cloudnull | clarkb corvus https://opendev.org/zuul/zuul-jobs/commit/8945377001c92b320787710c1caa5d25ddff8636 ? | 20:58 |
| *** rh-jelabarre has joined #openstack-infra | 20:59 | |
| *** kjackal has quit IRC | 21:00 | |
| clarkb | ianw: I had to manually install openafs from the ppa. The server in fortnebula had the 1.8.0 version installed and apt-get install openafs-client installed 1.8.3 after so still something up with the ansible I think | 21:02 |
| clarkb | ianw: also it seems that the cache size for afs is only 500MB on these nodes? I think that might be another bit of ansible we need to update? | 21:02 |
| *** goldyfruit has joined #openstack-infra | 21:02 | |
| *** ekultails has quit IRC | 21:02 | |
| *** weifan has joined #openstack-infra | 21:02 | |
| *** jtomasek has quit IRC | 21:03 | |
| clarkb | ok dkms rebuilt kernel modules, I rebooted and now we have a /afs. I thinkthe next ansible run should be able to finish configuring the mirror | 21:09 |
| fungi | yeah, http://zuul.opendev.org/t/zuul/buildsets?change=668954%2C5 isn't turning up anything and http://zuul.opendev.org/t/openstack/buildsets?change=668954%2C5 only goes up to ,4 | 21:09 |
| fungi | could it have been dequeued when the tenant changed? | 21:10 |
| clarkb | cloudnull: ya that may be it? | 21:10 |
| clarkb | fungi: I think it was rechecekd too | 21:10 |
| *** slaweq has quit IRC | 21:11 | |
| *** igordc has joined #openstack-infra | 21:14 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove absentee clouds from cloud launcher https://review.opendev.org/668992 | 21:15 |
| *** smarcet has joined #openstack-infra | 21:15 | |
| clarkb | I think ^ is necessary to fix the cloud launcher setup for fortnebula | 21:15 |
| *** pcaruana has quit IRC | 21:16 | |
| *** ijw has joined #openstack-infra | 21:20 | |
| *** rfolco has quit IRC | 21:22 | |
| *** slaweq has joined #openstack-infra | 21:27 | |
| *** e0ne has quit IRC | 21:28 | |
| *** e0ne has joined #openstack-infra | 21:29 | |
| *** slaweq has quit IRC | 21:32 | |
| clarkb | and now LE cert provisioning seems broken | 21:33 |
| *** Fidde has quit IRC | 21:34 | |
| clarkb | I know what the problem is | 21:38 |
| *** slaweq has joined #openstack-infra | 21:38 | |
| *** ijw has quit IRC | 21:38 | |
| *** ijw has joined #openstack-infra | 21:39 | |
| *** ijw has quit IRC | 21:39 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add fortnebula mirror LE details dict https://review.opendev.org/668998 | 21:40 |
| clarkb | infra-root ^ I think that is the last bit in getting a working mirror in fortnebula cloud | 21:40 |
| *** ijw has joined #openstack-infra | 21:40 | |
| corvus | ...working backwards on backscroll... | 21:40 |
| corvus | clarkb: +3 the LE change | 21:42 |
| clarkb | corvus: thanks | 21:43 |
| *** slaweq has quit IRC | 21:43 | |
| *** ijw has quit IRC | 21:44 | |
| corvus | looking at 668954 | 21:46 |
| *** eernst has quit IRC | 21:46 | |
| *** e0ne has quit IRC | 21:46 | |
| *** tdasilva has quit IRC | 21:50 | |
| *** eernst_ has joined #openstack-infra | 21:54 | |
| corvus | i think i see a zuul bug | 21:54 |
| *** eernst_ has quit IRC | 21:56 | |
| *** eernst has joined #openstack-infra | 21:56 | |
| *** rlandy is now known as rlandy|bbl | 21:58 | |
| *** eernst has quit IRC | 22:00 | |
| *** rcernin has quit IRC | 22:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: update docs for mirror-update.opendev.org https://review.opendev.org/668833 | 22:04 |
| *** ijw has joined #openstack-infra | 22:04 | |
| *** ijw has joined #openstack-infra | 22:05 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: export mirroring logs https://review.opendev.org/668832 | 22:06 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: update docs for mirror-update.opendev.org https://review.opendev.org/668833 | 22:06 |
| ianw | clarkb: ok, here's the problem with the openafs install -- we didn't merge the fix yet -> https://review.opendev.org/#/c/668745/ :) | 22:07 |
| ianw | the cache size ... hrm | 22:09 |
| *** Wryhder has joined #openstack-infra | 22:09 | |
| clarkb | ianw: aha | 22:09 |
| clarkb | ianw: ya we seem to use the default cache size from debconf which is 500MB | 22:09 |
| *** Lucas_Gray has quit IRC | 22:09 | |
| clarkb | ianw: re update docs for mirror-update I left a couple comments on ps2 | 22:09 |
| *** Wryhder is now known as Lucas_Gray | 22:10 | |
| clarkb | ianw: corvus https://review.opendev.org/#/c/668992/ is the other change I'm looking forward too if you get a chance from debugging zuul | 22:10 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/roles/openafs-client/defaults/main.yaml#L2 | 22:11 |
| ianw | so the knobs are there for turning, might have forgotten to actually turn them :) | 22:12 |
| *** smarcet has left #openstack-infra | 22:17 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: update docs for mirror-update.opendev.org https://review.opendev.org/668833 | 22:20 |
| openstackgerrit | Merged opendev/system-config master: Add fortnebula mirror LE details dict https://review.opendev.org/668998 | 22:25 |
| *** bobh has joined #openstack-infra | 22:31 | |
| *** tosky has quit IRC | 22:38 | |
| openstackgerrit | Merged opendev/system-config master: openafs-client: ensure latest package and reorder install https://review.opendev.org/668745 | 22:41 |
| *** whoami-rajat has quit IRC | 22:42 | |
| *** yamamoto has joined #openstack-infra | 22:45 | |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 23:02 |
| clarkb | bah I'm really bad at this LE thing | 23:02 |
| clarkb | ianw: I know most of us are about to disappear but maybe this needs docs? | 23:03 |
| ianw | clarkb: yeah, i was just going to write some :) what i'd actually like is a linter ... but i'm not sure how to implement that | 23:03 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add apache restart handler for fortnebula LE setup https://review.opendev.org/669007 | 23:05 |
| clarkb | I think ^ won't help get apache started now because the certs did provision so we won't try that again for 60 days | 23:05 |
| clarkb | so I'll go ahead and restart apache now | 23:05 |
| ianw | clarkb: hrm, did ansible not stop with that missing? it used to bail without a handler defined but never know ... that's the bit i wanted to lint | 23:06 |
| clarkb | ianw: it bailed but that happens after the cert is provisioned | 23:07 |
| clarkb | ianw: so once that chagne above merges it won't need to reprovision the cert and it won't start apache | 23:07 |
| clarkb | https://review.opendev.org/#/c/668992/ is the current blocker becuase security groups are such that you can't hit http(s) from outside of that security group | 23:07 |
| *** rcernin has joined #openstack-infra | 23:09 | |
| clarkb | and with that I may have reached my holidaying point. I'll likely be around friday to try and get the cloud running with nodepool and all that | 23:09 |
| *** owalsh_ has joined #openstack-infra | 23:10 | |
| *** owalsh has quit IRC | 23:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: export mirroring logs https://review.opendev.org/668832 | 23:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: update docs for mirror-update.opendev.org https://review.opendev.org/668833 | 23:11 |
| *** raissa has joined #openstack-infra | 23:13 | |
| *** slaweq has joined #openstack-infra | 23:14 | |
| *** Lucas_Gray has quit IRC | 23:14 | |
| *** slaweq has quit IRC | 23:19 | |
| *** mattw4 has quit IRC | 23:21 | |
| *** dchen has joined #openstack-infra | 23:27 | |
| *** weifan has quit IRC | 23:30 | |
| *** bobh has quit IRC | 23:32 | |
| *** bobh has joined #openstack-infra | 23:33 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix multi-tenant caching of extra config files https://review.opendev.org/669008 | 23:34 |
| corvus | clarkb, cloudnull, fungi: ^ that's why no jobs are running on zuul-jobs right now | 23:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix multi-tenant caching of extra config files https://review.opendev.org/669008 | 23:37 |
| *** bobh has quit IRC | 23:37 | |
| *** gyee has quit IRC | 23:39 | |
| *** nicolasbock has quit IRC | 23:41 | |
| *** bobh has joined #openstack-infra | 23:48 | |
| *** yamamoto has quit IRC | 23:48 | |
| *** gyee has joined #openstack-infra | 23:54 | |
| *** bobh has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
