| clarkb | ya seems like a reasonable compromise for staying on top of arm64 compat while managing the limited resources we have | 00:02 |
|---|---|---|
| ianw | for the static server, should we have one cert with multiple hostnames, or do a separate cert for each? does it matter? | 00:03 |
| clarkb | we are going to LE them right? I half expect we'd do a single cert for that? | 00:05 |
| clarkb | I guess the LE hostvars input is a list so could split them up? | 00:06 |
| clarkb | I guess I don't have a strong opinion | 00:06 |
| *** mattw4 has quit IRC | 00:07 | |
| ianw | yeah, we can issue different keys for governance.openstack.org/security.openstack.org, or put them in the same with SAN | 00:08 |
| clarkb | fungi: ^ any opinions? | 00:08 |
| *** smarcet has quit IRC | 00:15 | |
| clarkb | Ok I signed up for a geoguessr pro account so that I can make a shared challenge | 00:19 |
| clarkb | maybe tomorrow after the meeting we can all play a round of "whose geography/travel knowledge is best" | 00:19 |
| clarkb | It generates a url which I can share and then I think you can play without an account? I guess we'll see tomorrow | 00:20 |
| openstackgerrit | Merged opendev/system-config master: install-ansible: Pre-install build deps for aarch64 https://review.opendev.org/691187 | 00:22 |
| openstackgerrit | Merged opendev/system-config master: Add arm64 mirror test https://review.opendev.org/690798 | 00:22 |
| ianw | " | 00:37 |
| ianw | Regarding which approach is “better” - I’d generally recommend keeping subdomains of the same domain on the same certificate, and different domains on different certificates, to minimize the risk of hitting the rate limits." | 00:37 |
| clarkb | That makes sense | 00:37 |
| ianw | random internet comment ... https://community.letsencrypt.org/t/san-vs-sni-alternate-domain-visibility/53443# | 00:37 |
| ianw | but that was ~ what i was thinking | 00:37 |
| *** rlandy|bbl is now known as rlandy | 00:58 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: DNM test all the things on ansible 2.9 with default python https://review.opendev.org/698344 | 01:01 |
| fungi | ianw: clarkb: sorry for the delay, but honestly whichever model is simpler is fine. the certs are served from the same server either way so whether they share a key doesn't have any security implications | 01:13 |
| ianw | fungi: i think separate certs ... but i can't understand why testing i'm getting "AH02032: Hostname localhost provided via SNI and hostname security.openstack.org provided via HTTP have no compatible SSL setup" | 01:13 |
| fungi | and yeah, i wouldn't be surprised if a cross-domain cert gets complicated with letsencrypt dns validation | 01:14 |
| ianw | but i don't get the same doing the same thing for governance.openstack.org .. and they seem to be the same | 01:14 |
| *** goldyfruit has quit IRC | 01:14 | |
| fungi | that sounds like a configuration issue | 01:14 |
| fungi | "Hostname localhost provided via SNI" suggests you're testing against localhost? | 01:15 |
| ianw | yeah; https://review.opendev.org/#/c/697587/11/testinfra/test_static.py | 01:15 |
| ianw | the first works, the second doesn't ... weird | 01:15 |
| fungi | i suspect wget is stepping on your toes there | 01:16 |
| fungi | it's probably trying to do sni for "localhost" independent of the http header you're injecting | 01:16 |
| *** rosmaita has left #openstack-infra | 01:20 | |
| fungi | remember sni happens during (really before) ssl/tls handshake, so by the time you're passing http headers it's too late | 01:20 |
| ianw | hrm, but why doesn't it happen on the first request? | 01:20 |
| ianw | maybe it's reusing something | 01:20 |
| fungi | is the first request maybe winding up as the default vhost? | 01:20 |
| fungi | s/as/at/ | 01:20 |
| ianw | hrm ... possibly | 01:20 |
| fungi | have you tried overrides in /etc/hosts? or does that create different problems? | 01:22 |
| ianw | i might have to ... i think curl has that built into command-line | 01:22 |
| ianw | this worked for the mirror testing, but as you say, it might be because that's become the one-and-only vhost | 01:22 |
| *** yamamoto has joined #openstack-infra | 01:25 | |
| ianw | curl --insecure https://security.openstack.org --resolve security.openstack.org:443:127.0.0.1 might work better i think | 01:27 |
| *** goldyfruit has joined #openstack-infra | 01:29 | |
| *** gyee has quit IRC | 01:29 | |
| fungi | seems reasonable | 01:30 |
| *** larainema has joined #openstack-infra | 01:30 | |
| fungi | good to know curl has a --resolve | 01:31 |
| ianw | yeah and that wget is a bit of a trap | 01:32 |
| *** smarcet has joined #openstack-infra | 01:32 | |
| fungi | i'm definitely not finding any similar feature in wget | 01:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 01:34 |
| *** goldyfruit has quit IRC | 01:43 | |
| openstackgerrit | Merged opendev/puppet-ptgbot master: Deploy new logo.png and motd.js files https://review.opendev.org/694327 | 01:47 |
| *** jamesmcarthur has joined #openstack-infra | 01:53 | |
| *** Lucas_Gray has joined #openstack-infra | 02:01 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** rlandy has quit IRC | 02:08 | |
| ianw | nice that works | 02:10 |
| *** bnemec has quit IRC | 02:10 | |
| *** bnemec has joined #openstack-infra | 02:12 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** Goneri has quit IRC | 02:21 | |
| fungi | excellent! | 02:21 |
| *** jamesmcarthur has joined #openstack-infra | 02:38 | |
| *** yamamoto has joined #openstack-infra | 02:38 | |
| *** rfolco|bbl has quit IRC | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| openstackgerrit | Antony Messerli proposed openstack/diskimage-builder master: Adds support for Fedora 31 https://review.opendev.org/697713 | 02:56 |
| *** ricolin has joined #openstack-infra | 03:03 | |
| *** apetrich has quit IRC | 03:09 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| *** ykarel|away has joined #openstack-infra | 03:46 | |
| *** jamesmcarthur has joined #openstack-infra | 03:54 | |
| *** goldyfruit has joined #openstack-infra | 04:01 | |
| *** Lucas_Gray has quit IRC | 04:17 | |
| *** jamesmcarthur has quit IRC | 04:26 | |
| *** ykarel|away has quit IRC | 04:32 | |
| *** yamamoto has quit IRC | 04:34 | |
| *** yamamoto has joined #openstack-infra | 04:40 | |
| *** ykarel|away has joined #openstack-infra | 04:52 | |
| *** ykarel|away is now known as ykarel | 04:53 | |
| *** goldyfruit has quit IRC | 05:00 | |
| *** rh-jelabarre has quit IRC | 05:00 | |
| *** factor has quit IRC | 05:05 | |
| *** ramishra has joined #openstack-infra | 05:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use olso to parse size strings https://review.opendev.org/503574 | 05:20 |
| *** yamamoto has quit IRC | 05:20 | |
| *** raukadah is now known as chkumar|rover | 05:33 | |
| *** factor has joined #openstack-infra | 05:34 | |
| *** slaweq has joined #openstack-infra | 05:36 | |
| *** yamamoto has joined #openstack-infra | 05:41 | |
| *** slaweq has quit IRC | 05:43 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Update nodepool dib stats https://review.opendev.org/638583 | 05:52 |
| *** yamamoto has quit IRC | 05:53 | |
| *** yamamoto has joined #openstack-infra | 05:54 | |
| *** yamamoto has quit IRC | 06:07 | |
| *** ricolin_ has joined #openstack-infra | 06:10 | |
| openstackgerrit | Merged openstack/project-config master: Update nodepool dib stats https://review.opendev.org/638583 | 06:10 |
| *** ricolin has quit IRC | 06:13 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Install rng-tools in Red Hat family distro images https://review.opendev.org/697183 | 06:25 |
| openstackgerrit | Merged zuul/nodepool master: Dockerfile: install sudo for nodepool-builder https://review.opendev.org/694709 | 06:26 |
| openstackgerrit | Merged zuul/nodepool master: Dockerfile: add DEBUG environment flag https://review.opendev.org/694845 | 06:27 |
| *** jamesmcarthur has joined #openstack-infra | 06:28 | |
| *** jamesmcarthur has quit IRC | 06:33 | |
| *** hwoarang has quit IRC | 06:36 | |
| *** hwoarang has joined #openstack-infra | 06:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: DNM: Remove openstack-python-jobs-trusty https://review.opendev.org/699348 | 06:42 |
| *** yamamoto has joined #openstack-infra | 06:47 | |
| *** lpetrut has joined #openstack-infra | 06:48 | |
| *** yamamoto has quit IRC | 06:58 | |
| *** kozhukalov has joined #openstack-infra | 06:59 | |
| *** ricolin_ is now known as ricolin | 07:00 | |
| AJaeger | fungi, do you know whether we still need to run legacy-storyboard-js-integration-ubuntu-trusty and legacy-storyboard-js-integration jobs on storyboard and storyboard-webclient? Those are experimental. Can they get removed? | 07:06 |
| *** pkopec has joined #openstack-infra | 07:09 | |
| diablo_rojo_phon | AJaeger: I want to way they can get removed, but I would definitely wait for fungi's answer. | 07:09 |
| diablo_rojo_phon | *say | 07:09 |
| AJaeger | thanks, diablo_rojo_phon. In that case let me push a change and wait for fungi to review ;) | 07:12 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove experimental legacy-storyboard-integration jobs https://review.opendev.org/699353 | 07:14 |
| AJaeger | fungi, diablo_rojo_phon ^ - I'll WIP until fungi confirms. | 07:14 |
| diablo_rojo_phon | Hopefully I'm not wrong lol | 07:20 |
| AJaeger | it's a tiny change, so no hazzle if you are ;) | 07:35 |
| AJaeger | hazzle? Trouble? Argh, my English is rusty ;( | 07:35 |
| *** ykarel is now known as ykarel|lunch | 07:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove legacy-group-based-policy trusty jobs https://review.opendev.org/699360 | 07:47 |
| *** pcaruana has joined #openstack-infra | 07:52 | |
| *** lpetrut has quit IRC | 07:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove trusty fuel jobs https://review.opendev.org/699362 | 07:54 |
| AJaeger | fungi, clarkb, ianw, you discussed trusty jobs, above are a few changes to remove most of them. | 07:55 |
| AJaeger | It still leaves a few in: legacy-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty, project-config-bindep-fallback-ubuntu-trusty, publish-wheel-mirror-ubuntu-trusty, legacy-gearman-plugin-maven-build-ubuntu-trusty, legacy-group-based-policy-dsvm-functional-ubuntu-trusty, legacy-zmq-event-publisher-maven-build-ubuntu-trusty. | 07:56 |
| AJaeger | Which of these are obsolete as well? | 07:56 |
| AJaeger | there's also legacy-logstash-filters-ubuntu-trusty | 07:58 |
| *** slaweq has joined #openstack-infra | 07:59 | |
| *** pgaxatte has joined #openstack-infra | 07:59 | |
| *** dchen has quit IRC | 08:03 | |
| openstackgerrit | Andreas Jaeger proposed opendev/storyboard master: Remove project stanza from zuul.yaml https://review.opendev.org/699363 | 08:04 |
| *** rpittau|afk is now known as rpittau | 08:05 | |
| *** yamamoto has joined #openstack-infra | 08:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: DNM: Remove openstack-python-jobs-trusty https://review.opendev.org/699348 | 08:14 |
| *** tkajinam has quit IRC | 08:18 | |
| *** tosky has joined #openstack-infra | 08:20 | |
| *** tesseract has joined #openstack-infra | 08:25 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| *** hashar has joined #openstack-infra | 08:36 | |
| *** yamamoto has quit IRC | 08:41 | |
| *** yamamoto has joined #openstack-infra | 08:46 | |
| *** ralonsoh has joined #openstack-infra | 08:46 | |
| *** apetrich has joined #openstack-infra | 08:47 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** lucasagomes has joined #openstack-infra | 08:54 | |
| *** priteau has joined #openstack-infra | 08:57 | |
| *** dklyle_ has joined #openstack-infra | 09:01 | |
| *** priteau has quit IRC | 09:02 | |
| *** kevinz_ has joined #openstack-infra | 09:02 | |
| *** dtantsur has joined #openstack-infra | 09:02 | |
| *** clayg_ has joined #openstack-infra | 09:03 | |
| *** priteau has joined #openstack-infra | 09:03 | |
| *** ildikov_ has joined #openstack-infra | 09:03 | |
| *** rpioso_ has joined #openstack-infra | 09:03 | |
| *** fresta_ has joined #openstack-infra | 09:03 | |
| *** davecore_ has joined #openstack-infra | 09:03 | |
| *** knikolla_ has joined #openstack-infra | 09:03 | |
| *** coreycb_ has joined #openstack-infra | 09:03 | |
| *** ykarel_ has joined #openstack-infra | 09:03 | |
| *** petevg_ has joined #openstack-infra | 09:03 | |
| *** tosky_ has joined #openstack-infra | 09:04 | |
| *** cyberpear_ has joined #openstack-infra | 09:04 | |
| *** zxiiro_ has joined #openstack-infra | 09:04 | |
| *** jistr_ has joined #openstack-infra | 09:05 | |
| *** benj_- has joined #openstack-infra | 09:05 | |
| *** Anticime1 has joined #openstack-infra | 09:05 | |
| *** iokiwi3 has joined #openstack-infra | 09:06 | |
| *** StevenK_ has joined #openstack-infra | 09:06 | |
| *** lpetrut has joined #openstack-infra | 09:06 | |
| *** ktsuyuzaki has joined #openstack-infra | 09:09 | |
| *** amotoki_ has joined #openstack-infra | 09:09 | |
| *** tesseract has quit IRC | 09:10 | |
| *** tosky has quit IRC | 09:10 | |
| *** kozhukalov has quit IRC | 09:10 | |
| *** ykarel has quit IRC | 09:10 | |
| *** psachin has quit IRC | 09:10 | |
| *** iokiwi has quit IRC | 09:10 | |
| *** rcernin has quit IRC | 09:10 | |
| *** szaher has quit IRC | 09:10 | |
| *** osmanlicilegi has quit IRC | 09:10 | |
| *** tbarron has quit IRC | 09:10 | |
| *** zer0c00l has quit IRC | 09:10 | |
| *** jistr has quit IRC | 09:10 | |
| *** benj_ has quit IRC | 09:10 | |
| *** AJaeger has quit IRC | 09:10 | |
| *** david-lyle has quit IRC | 09:10 | |
| *** SotK has quit IRC | 09:10 | |
| *** kambiz has quit IRC | 09:10 | |
| *** jamesdenton has quit IRC | 09:10 | |
| *** andreykurilin has quit IRC | 09:10 | |
| *** fnordahl has quit IRC | 09:10 | |
| *** StevenK has quit IRC | 09:10 | |
| *** cmorpheus has quit IRC | 09:10 | |
| *** dtantsur|afk has quit IRC | 09:10 | |
| *** kevinz has quit IRC | 09:10 | |
| *** markmcclain has quit IRC | 09:10 | |
| *** zxiiro has quit IRC | 09:10 | |
| *** admcleod has quit IRC | 09:10 | |
| *** calebb has quit IRC | 09:10 | |
| *** yankcrime has quit IRC | 09:10 | |
| *** radez has quit IRC | 09:10 | |
| *** nicholas has quit IRC | 09:10 | |
| *** arif-ali has quit IRC | 09:10 | |
| *** rakhmerov has quit IRC | 09:10 | |
| *** fresta has quit IRC | 09:10 | |
| *** rpioso has quit IRC | 09:10 | |
| *** kota_ has quit IRC | 09:10 | |
| *** cyberpear has quit IRC | 09:10 | |
| *** petevg has quit IRC | 09:10 | |
| *** SergeyLukjanov has quit IRC | 09:10 | |
| *** bradm has quit IRC | 09:10 | |
| *** amotoki has quit IRC | 09:10 | |
| *** zaro has quit IRC | 09:10 | |
| *** Anticimex has quit IRC | 09:10 | |
| *** coreycb has quit IRC | 09:10 | |
| *** clayg has quit IRC | 09:10 | |
| *** davecore has quit IRC | 09:10 | |
| *** knikolla has quit IRC | 09:10 | |
| *** ildikov has quit IRC | 09:10 | |
| *** kevinz_ is now known as kevinz | 09:12 | |
| *** clayg_ is now known as clayg | 09:12 | |
| *** petevg_ is now known as petevg | 09:12 | |
| *** rpioso_ is now known as rpioso | 09:12 | |
| *** szaher has joined #openstack-infra | 09:12 | |
| *** zxiiro_ is now known as zxiiro | 09:12 | |
| *** davecore_ is now known as davecore | 09:12 | |
| *** knikolla_ is now known as knikolla | 09:12 | |
| *** coreycb_ is now known as coreycb | 09:12 | |
| *** ildikov_ is now known as ildikov | 09:12 | |
| *** cyberpear_ is now known as cyberpear | 09:12 | |
| *** iokiwi3 is now known as iokiwi | 09:12 | |
| *** benj_- is now known as benj_ | 09:12 | |
| *** admcleod has joined #openstack-infra | 09:12 | |
| *** SotK has joined #openstack-infra | 09:12 | |
| *** yamamoto has quit IRC | 09:13 | |
| *** AJaeger has joined #openstack-infra | 09:16 | |
| *** zaro has joined #openstack-infra | 09:17 | |
| *** tesseract has joined #openstack-infra | 09:17 | |
| *** rcernin has joined #openstack-infra | 09:18 | |
| *** yamamoto has joined #openstack-infra | 09:18 | |
| *** tosky_ is now known as tosky | 09:19 | |
| *** ykarel_ is now known as ykarel | 09:20 | |
| *** yamamoto has quit IRC | 09:23 | |
| *** derekh has joined #openstack-infra | 09:33 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:34 | |
| *** arif-ali has joined #openstack-infra | 10:09 | |
| *** yankcrime has joined #openstack-infra | 10:15 | |
| *** hashar has quit IRC | 10:24 | |
| *** priteau has quit IRC | 10:38 | |
| *** ociuhandu has joined #openstack-infra | 10:49 | |
| *** kozhukalov has joined #openstack-infra | 10:56 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:06 | |
| *** ijw has joined #openstack-infra | 11:07 | |
| *** ijw has quit IRC | 11:12 | |
| *** lucasagomes has quit IRC | 11:24 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** ociuhandu has joined #openstack-infra | 11:28 | |
| *** yamamoto has quit IRC | 11:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove openstack-python-jobs-trusty template https://review.opendev.org/699348 | 11:32 |
| *** ociuhandu has quit IRC | 11:32 | |
| *** ociuhandu has joined #openstack-infra | 11:33 | |
| *** osmanlicilegi has joined #openstack-infra | 11:37 | |
| *** ianychoi has quit IRC | 11:38 | |
| *** ociuhandu has quit IRC | 11:39 | |
| *** ianychoi has joined #openstack-infra | 11:40 | |
| *** surpatil has joined #openstack-infra | 11:41 | |
| *** SurajPatil has joined #openstack-infra | 11:41 | |
| *** yamamoto has joined #openstack-infra | 11:41 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** smarcet has quit IRC | 11:52 | |
| *** yamamoto has joined #openstack-infra | 11:54 | |
| *** ociuhandu has joined #openstack-infra | 11:55 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** rfolco|bbl has joined #openstack-infra | 11:57 | |
| Shrews | AJaeger: I like this new word "hazzle" that you have invented. | 11:59 |
| *** amotoki_ is now known as amotoki | 12:12 | |
| openstackgerrit | Merged zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 12:19 |
| *** rfolco|bbl is now known as rfolco | 12:27 | |
| *** rosmaita has joined #openstack-infra | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:37 | |
| *** tesseract has quit IRC | 12:38 | |
| *** tesseract has joined #openstack-infra | 12:38 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** lucasagomes has joined #openstack-infra | 12:39 | |
| *** goldyfruit has joined #openstack-infra | 12:41 | |
| *** hwoarang has quit IRC | 12:53 | |
| *** hwoarang has joined #openstack-infra | 12:53 | |
| *** gfidente has joined #openstack-infra | 12:54 | |
| *** surpatil has quit IRC | 12:54 | |
| *** SurajPatil has quit IRC | 12:55 | |
| *** surpatil has joined #openstack-infra | 12:55 | |
| *** SurajPatil has joined #openstack-infra | 12:55 | |
| *** jamesmcarthur has joined #openstack-infra | 12:56 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:58 | |
| *** rlandy has joined #openstack-infra | 12:59 | |
| *** SurajPatil has quit IRC | 13:00 | |
| *** SurajPatil has joined #openstack-infra | 13:01 | |
| *** surpatil has quit IRC | 13:01 | |
| *** cmurphy has joined #openstack-infra | 13:02 | |
| *** surpatil has joined #openstack-infra | 13:02 | |
| *** jamesmcarthur has quit IRC | 13:02 | |
| *** jamesmcarthur has joined #openstack-infra | 13:08 | |
| *** lbragstad_ is now known as lbragstad | 13:08 | |
| *** rh-jelabarre has joined #openstack-infra | 13:11 | |
| *** udesale has joined #openstack-infra | 13:11 | |
| *** ykarel is now known as ykarel|afk | 13:12 | |
| *** udesale has quit IRC | 13:12 | |
| *** udesale has joined #openstack-infra | 13:13 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use explicit image paths https://review.opendev.org/690511 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update pip3 role to install from get-pip.py https://review.opendev.org/690766 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service playbook and test run for prod gerrit https://review.opendev.org/691171 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add launchpadlib credentials to gerrit ansible https://review.opendev.org/691172 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add replication config to gerrit ansible https://review.opendev.org/691173 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through storyboard hiera data https://review.opendev.org/691777 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Clean up review comments https://review.opendev.org/692003 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 13:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update bazel to version 1.2.0 https://review.opendev.org/699406 | 13:13 |
| mordred | Shrews: do you see how good I am at being out? | 13:13 |
| Shrews | mordred: it's about what I expected, tbh | 13:15 |
| *** ykarel|afk has quit IRC | 13:17 | |
| *** smarcet has joined #openstack-infra | 13:27 | |
| *** ociuhandu has quit IRC | 13:28 | |
| *** ociuhandu has joined #openstack-infra | 13:28 | |
| AJaeger | fungi, thanks for confirming the storyboard jobs! | 13:30 |
| *** jamesmcarthur has quit IRC | 13:31 | |
| mordred | fungi: got a sec and feel like +Aing a python2 remove patch? https://review.opendev.org/#/c/699132/ | 13:31 |
| AJaeger | mordred: want to +2A https://review.opendev.org/699363 and https://review.opendev.org/699353, please? | 13:33 |
| mordred | ++ | 13:34 |
| AJaeger | thanks | 13:34 |
| *** ociuhandu has quit IRC | 13:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-storyboard-js-integration jobs https://review.opendev.org/699412 | 13:36 |
| AJaeger | mordred, fungi, and followup cleanup, please ^ | 13:36 |
| openstackgerrit | Monty Taylor proposed opendev/storyboard master: Add a dedicated "lpimport" tox testenv https://review.opendev.org/649065 | 13:37 |
| *** udesale has quit IRC | 13:39 | |
| *** jpena|lunch is now known as jpena | 13:41 | |
| *** jamesmcarthur has joined #openstack-infra | 13:42 | |
| mordred | fungi: thanks! | 13:46 |
| mordred | AJaeger: I assume you'll recheck that one when appropriate | 13:46 |
| openstackgerrit | Merged openstack/project-config master: Remove experimental legacy-storyboard-integration jobs https://review.opendev.org/699353 | 13:47 |
| AJaeger | mordred: just done - thanks | 13:48 |
| *** yamamoto has quit IRC | 13:49 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| *** yamamoto has quit IRC | 13:51 | |
| *** mriedem has joined #openstack-infra | 13:57 | |
| openstackgerrit | Merged opendev/storyboard master: Remove project stanza from zuul.yaml https://review.opendev.org/699363 | 13:58 |
| *** smarcet has quit IRC | 14:00 | |
| *** dpawlik has joined #openstack-infra | 14:00 | |
| *** ociuhandu has joined #openstack-infra | 14:01 | |
| *** smarcet has joined #openstack-infra | 14:01 | |
| *** sshnaidm|afk is now known as sshnaidm | 14:02 | |
| *** liuyulong has joined #openstack-infra | 14:03 | |
| *** tkajinam has joined #openstack-infra | 14:04 | |
| *** smarcet has quit IRC | 14:05 | |
| *** surpatil has quit IRC | 14:12 | |
| *** SurajPatil has quit IRC | 14:13 | |
| *** smarcet has joined #openstack-infra | 14:14 | |
| *** ociuhandu has quit IRC | 14:21 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy-storyboard-js-integration jobs https://review.opendev.org/699412 | 14:22 |
| *** smarcet has quit IRC | 14:27 | |
| *** smarcet has joined #openstack-infra | 14:31 | |
| rm_work | trying to run devstack on cent7 and it's exploding during a package install step because debootstrap isn't found (obviously)... am i missing something obvious in my config? pretty sure this used to work before <_< | 14:36 |
| *** factor has quit IRC | 14:39 | |
| *** eharney has joined #openstack-infra | 14:42 | |
| mnaser | rm_work: i think debootstrap is in epel | 14:42 |
| mnaser | so i would add it and you'll probably be ok | 14:42 |
| *** addyess has quit IRC | 14:42 | |
| *** smarcet has quit IRC | 14:43 | |
| *** panda has quit IRC | 14:44 | |
| *** panda has joined #openstack-infra | 14:44 | |
| *** tkajinam has quit IRC | 14:47 | |
| *** ykarel|afk has joined #openstack-infra | 14:47 | |
| *** addyess has joined #openstack-infra | 14:47 | |
| *** smarcet has joined #openstack-infra | 14:48 | |
| rm_work | ah ok | 14:50 |
| rm_work | didn't think you could do that on an RPM distro :D | 14:50 |
| *** ykarel|afk is now known as ykarel | 14:51 | |
| mnaser | :) | 14:51 |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| *** yamamoto has quit IRC | 14:54 | |
| fungi | just like you an use rpm on a debian derivative | 14:54 |
| fungi | er, you can | 14:54 |
| *** yamamoto has joined #openstack-infra | 14:55 | |
| *** yamamoto has quit IRC | 15:00 | |
| *** smarcet has quit IRC | 15:01 | |
| *** ykarel has quit IRC | 15:03 | |
| *** ykarel has joined #openstack-infra | 15:03 | |
| *** smarcet has joined #openstack-infra | 15:05 | |
| rm_work | blegh, we have a custom internal epel repo and it's missing debootstrap, that's why this is failing lol | 15:06 |
| *** kozhukalov has quit IRC | 15:07 | |
| fungi | infra-root: heads up, we got another report of an exposed portmapper socket on a job node in inap, their scanner saw it on 198.72.124.78 at 2019-12-12T00:00:45.215319 utc | 15:08 |
| *** piotrowskim has joined #openstack-infra | 15:09 | |
| fungi | i'm a bit strapped for time trying to prepare to head out of town tomorrow, does anyone have a few minutes to track it back to a build and correlate the findings with my analysis from a week ago? http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-12-10.log.html#t2019-12-10T15:21:53 | 15:09 |
| *** lourot has quit IRC | 15:10 | |
| *** zzehring has joined #openstack-infra | 15:12 | |
| *** ociuhandu has joined #openstack-infra | 15:14 | |
| corvus | i'll track it back to a build | 15:16 |
| *** lourot has joined #openstack-infra | 15:16 | |
| *** ociuhandu has quit IRC | 15:18 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: 'keys' must be defined for host-key-checking: false https://review.opendev.org/698029 | 15:19 |
| corvus | fungi, mgagne: it looks like we did not have that IP at that time. here are the log entries for the node before and the node after: http://paste.openstack.org/show/787677/ | 15:23 |
| *** jaosorior has joined #openstack-infra | 15:26 | |
| *** smarcet has quit IRC | 15:36 | |
| *** ociuhandu has joined #openstack-infra | 15:36 | |
| *** eharney has quit IRC | 15:43 | |
| *** michael-beaver has joined #openstack-infra | 15:45 | |
| *** ociuhandu has quit IRC | 15:46 | |
| *** jamesmcarthur has quit IRC | 15:47 | |
| *** jamesmcarthur has joined #openstack-infra | 15:48 | |
| *** chkumar|rover is now known as raukadah | 15:50 | |
| *** jamesmcarthur has quit IRC | 15:53 | |
| *** smarcet has joined #openstack-infra | 15:55 | |
| *** jamesmcarthur has joined #openstack-infra | 15:59 | |
| *** lucasagomes has quit IRC | 16:00 | |
| fungi | thanks corvus! i've definitely seen that happen with their scans before as well, they don't seem to strictly correlate time in their nova logs (or make mistakes when doing so). maybe they assume servers which show up in their scans are long-lived and so just look to see which tenant had that ip address most recently or currently? | 16:01 |
| *** ykarel is now known as ykarel|away | 16:04 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: DNM test all the things on ansible 2.9 with default python https://review.opendev.org/698344 | 16:04 |
| *** hwoarang has quit IRC | 16:06 | |
| *** ociuhandu has joined #openstack-infra | 16:07 | |
| *** larainema has quit IRC | 16:09 | |
| *** hwoarang has joined #openstack-infra | 16:11 | |
| *** ociuhandu has quit IRC | 16:12 | |
| AJaeger | config-core, working on trusty removal: https://review.opendev.org/699348 removes a unused template; https://review.opendev.org/699362 and https://review.opendev.org/699360 remove some jobs on repos/branches that are not used. Reviews welcome! | 16:19 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Use present for package state instead of installed https://review.opendev.org/699450 | 16:24 |
| fungi | just discovered in #openstack-nova, devstack-gate defaults to installing ansible 2.5 | 16:27 |
| *** ociuhandu has joined #openstack-infra | 16:27 | |
| clarkb | fungi: ya, because everytime ansible would update it would break the gate | 16:28 |
| fungi | which is no longer compatible with ara, so grenade jobs are blowing up because grenade still uses devstack-gate | 16:28 |
| clarkb | so we pinned it | 16:28 |
| clarkb | ah I guess we need to pin ara too then | 16:28 |
| fungi | yeah, mriedem ^ that's another option | 16:28 |
| clarkb | we can also bump the pin for ansible up | 16:29 |
| clarkb | and that chnge should be self testing | 16:29 |
| clarkb | 2.7 is likely a safe ish jump | 16:29 |
| *** gyee has joined #openstack-infra | 16:29 | |
| clarkb | 2.8 and 2.9 less safe | 16:29 |
| mriedem | looks like we do pin ara https://github.com/openstack/devstack-gate/blob/13972f3a9b40b03121b7e8c6270cae36cf947beb/devstack-vm-gate-wrap.sh#L499 | 16:30 |
| clarkb | huh did ara < 1.0.0 get releases that stopped supporting old ansible? | 16:31 |
| clarkb | I think we should start by pushing a change to try ansible 2.7 | 16:31 |
| clarkb | that I expect will be mostly safe | 16:31 |
| mriedem | https://pypi.org/project/ara/#history | 16:32 |
| *** ociuhandu has quit IRC | 16:32 | |
| openstackgerrit | Antony Messerli proposed openstack/diskimage-builder master: Adds support for Fedora 31 https://review.opendev.org/697713 | 16:32 |
| mriedem | yeah ara 0.16.6 dropped support for ansible < 2.6 | 16:36 |
| clarkb | but that was back in november? | 16:36 |
| mriedem | yup | 16:36 |
| mriedem | not sure why it's showing up now | 16:36 |
| clarkb | weird. But ya I think lets try ansible 2.7 and if that fails we can pin ara < 0.16.6 | 16:37 |
| mriedem | https://github.com/ansible-community/ara/commit/872ba818fbc5267b1f769d5485fd3ab318235aea fwiw | 16:37 |
| openstackgerrit | Merged zuul/nodepool master: Add container-with-siblings functional test https://review.opendev.org/693464 | 16:37 |
| clarkb | mriedem: do you want to push that change or should I/ | 16:37 |
| mriedem | i can push it | 16:37 |
| mriedem | just making sure we have ansible 2.7 on xenial nodes and we do: | 16:40 |
| mriedem | 2019-12-06 01:06:16.927660 | Ansible Version: 2.7.14 | 16:40 |
| *** pgaxatte has quit IRC | 16:40 | |
| mriedem | ^ is from a stable/pike grenade job | 16:40 |
| *** eharney has joined #openstack-infra | 16:43 | |
| *** ricolin has quit IRC | 16:44 | |
| *** lpetrut has quit IRC | 16:44 | |
| *** jpena is now known as jpena|brb | 16:45 | |
| mriedem | weird - looking at an ocata tempest job that's using xenial it is using ansible 2.8.7, but maybe because it's using the uCA | 16:47 |
| mriedem | *UCA | 16:47 |
| *** jamesmcarthur has quit IRC | 16:48 | |
| *** jamesmcarthur has joined #openstack-infra | 16:48 | |
| openstackgerrit | Matt Riedemann proposed openstack/devstack-gate master: Bump ANSIBLE_VERSION default to 2.7.14 https://review.opendev.org/699463 | 16:51 |
| clarkb | mriedem: there are two layers of ansible here too. The zuul ansible (whihc should be 2.8.7) and the ansibel that d-g runs under the zuul ansible which is currently 2.5 or so | 16:52 |
| *** jamesmcarthur has quit IRC | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:53 | |
| *** ociuhandu has joined #openstack-infra | 16:53 | |
| *** efried_pto has quit IRC | 16:54 | |
| *** ociuhandu has quit IRC | 16:55 | |
| *** ociuhandu has joined #openstack-infra | 16:55 | |
| mriedem | it's also funny that this appears to be intermittent | 16:55 |
| mriedem | i have a nova change on master that just got into the gate | 16:55 |
| clarkb | perhaps the ara compatibility depends on ansible's run state which depends on factors that aren't consistent? | 16:56 |
| *** jamesmcarthur has quit IRC | 16:56 | |
| clarkb | some runs it "just works" by chance? | 16:56 |
| mriedem | yeah idk, looking at logstash it's multiple branches/jobs/providers | 16:59 |
| *** ykarel|away has quit IRC | 16:59 | |
| *** tesseract has quit IRC | 17:00 | |
| mriedem | so, it might not be the ansible thing | 17:02 |
| mriedem | https://zuul.opendev.org/t/openstack/build/ce17ff84afb04b339cb98917c3625ced/log/logs/devstack-gate-setup-host.txt#1847 | 17:02 |
| mriedem | in this case it was a network failure | 17:02 |
| openstackgerrit | Merged zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 17:06 |
| mriedem | clarkb: why are we even installing ara anymore? looking at a tempest job on master it doesn't use ara - i thought the zuul console stuff replaced ara? | 17:06 |
| dmsimard | ara 0.x /probably/ still works with 2.5 but it's untested | 17:06 |
| clarkb | mriedem: its for the nested ansible run. You are correct that the zuul level ansible doesn't use ara anymore | 17:07 |
| mriedem | and ara from a grenade/devstack-gate job is busted: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ed7/638047/56/check/nova-grenade-multinode/ed76ba4/logs/ara/index.html.gz | 17:07 |
| mriedem | ok so maybe ^ is just the result of the issue we're trying to fix | 17:07 |
| dmsimard | mriedem: I'm not sure gzipping the reports work without a web server in front to transparently rewrite URLs and decompress things | 17:07 |
| dmsimard | uncompressed reports should just work | 17:08 |
| clarkb | dmsimard: the file encodings should be set properly in swift | 17:08 |
| clarkb | so your browser should decompress them | 17:08 |
| *** smarcet has left #openstack-infra | 17:08 | |
| dmsimard | clarkb: sure, but the links to the css/js files are not the compressed versions | 17:08 |
| clarkb | ah | 17:09 |
| dmsimard | the logserver had a rule to try the gzipped version first and fall back to uncompressed | 17:09 |
| clarkb | then the fix would be to stop having an explicit compression step in the job and let the upload manage it | 17:09 |
| clarkb | we compress on upload if not already compressed, but keep the filename and set the encoding | 17:09 |
| dmsimard | for ara-report specifically, there is a toggle for that: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ara-report/tasks/main.yaml#L61 | 17:10 |
| dmsimard | if there is something else that goes and compress everything, it would override that though | 17:10 |
| mriedem | btw, if there is a goal to drop devstack-gate usage grenade jobs are probably the big remaining user of d-g and https://review.opendev.org/#/c/548936/102 is the thing holding that back | 17:14 |
| mriedem | i haven't been following it but it's huge and scary | 17:15 |
| clarkb | mriedem: yup. tosky has been working on grenade jobs without d-g | 17:15 |
| mgagne | corvus, fungi: IP addresses aren't shared at inap. This means no other costumer can use them beside your account. | 17:18 |
| clarkb | maybe the timestamp is wrong then? | 17:19 |
| tosky | clarkb, mriedem the jobs seems to be working; there is just one comment by Sean Mooney about a possible improvement | 17:19 |
| tosky | and I hold back waiting for more comments | 17:19 |
| tosky | I guess I need to start working again on them | 17:19 |
| clarkb | tosky: we might want to land that during the holiday quiet period | 17:19 |
| clarkb | worst case it breaks and we can revert? | 17:20 |
| *** Goneri has joined #openstack-infra | 17:20 | |
| clarkb | but may give us a low volume time to burn it in a bit | 17:20 |
| tosky | if you want to comment: https://review.opendev.org/#/q/topic:grenade_zuulv3+status:open | 17:20 |
| clarkb | tosky: I'll try to make time for it after the infra meeting today | 17:20 |
| tosky | thanks | 17:21 |
| tosky | if that improvement is the only request that comes out, I will try to implement it as soon as possible | 17:21 |
| tosky | basically everything seemed to be working, apart from a multinode grenade job which failed while running their tests I think after the upgrade | 17:22 |
| tosky | but not sure it was really related to my changes | 17:22 |
| tosky | aaaanyway, there is a long history of comments :) | 17:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use present for package state instead of installed https://review.opendev.org/699450 | 17:22 |
| *** jpena|brb is now known as jpena | 17:23 | |
| tosky | uh, weird (the author) | 17:23 |
| *** ramishra has quit IRC | 17:23 | |
| tosky | let me restore it | 17:23 |
| fungi | mgagne: what are the chances future reports could include a nova instance name or uuid? i suppose dereferencing ip addresses in real time during the scan is nontrivial? | 17:23 |
| AJaeger | mriedem: there's a goal to use "zuul v3 native jobs" and that implies getting rid of devstack-gate. Community goal for V release, did not make it for U | 17:23 |
| fungi | mainly because of concerns grenade wouldn't be updated in time for ussuri | 17:24 |
| mgagne | fungi: little to no chance. The scanning system isn't linked to OpenStack. It is performed by a different department which scans all our IP space, including non-openstack systems. | 17:24 |
| *** rpittau is now known as rpittau|afk | 17:25 | |
| fungi | mgagne: no worries, figured it didn't hurt to ask. i can understand that wouldn't be easy. i also wonder if that address could belong to a rogue vm nova has lost track of? | 17:25 |
| mgagne | fungi: is it the same IP as last time? I didn't check. | 17:26 |
| mgagne | fungi: if it's rogue, you should be able to ping it right now. | 17:26 |
| fungi | it's not the same ip address as a week prior, no, i just checked that myself too | 17:27 |
| fungi | that ip address is pingable at the moment | 17:27 |
| fungi | 198.72.124.78 | 17:27 |
| fungi | but it may have a running node i'll check | 17:27 |
| fungi | yeah, up just over an hour | 17:28 |
| fungi | LSB Version: :core-4.1-amd64:core-4.1-noarch | 17:29 |
| fungi | do we have coreos nodes? | 17:30 |
| fungi | is that what that means? | 17:30 |
| mgagne | fungi: I'll try to ask for more details | 17:30 |
| mgagne | fungi: about the timestamp | 17:30 |
| fungi | thanks | 17:30 |
| tosky | mriedem, clarkb : fixed the attribution of one review, rebase all the stack | 17:31 |
| tosky | rebased* | 17:31 |
| *** jamesmcarthur has joined #openstack-infra | 17:31 | |
| fungi | yeah, at the moment 198.72.124.78 is in use booted as a centos-7 job node | 17:32 |
| fungi | so definitely explains why it's pingable | 17:32 |
| clarkb | fungi: jobs can replace their root disk and reboot | 17:32 |
| fungi | yeah, i think this is an octavia job which has done just that | 17:33 |
| *** jamesmcarthur has quit IRC | 17:33 | |
| fungi | judging from the ansible processes i see running | 17:33 |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| johnsom | fungi What did we do now? lol | 17:35 |
| fungi | nothing! | 17:35 |
| johnsom | Oh, ok, something I can help with then? | 17:35 |
| fungi | i was just logging into a node to check that it was doing stuff, and the motd and lsb_release output confused me | 17:35 |
| fungi | it's not a problem | 17:36 |
| clarkb | unless coreos opens a bunch of servies to the internet without a firewall | 17:36 |
| johnsom | We don't have any coreos jobs (we don't support it) | 17:36 |
| fungi | clarkb: yeah, i don't think the running build is in any way related to the notification from a week ago | 17:37 |
| johnsom | I looked at it about a year ago and it seemed like it was going to be a bunch of work to get the image size down | 17:37 |
| fungi | (and i did check `netstat -lnu` for 111/udp just out of curiosity while i was in there) | 17:37 |
| fungi | aha, it's a tripleo job | 17:38 |
| *** mattw4 has joined #openstack-infra | 17:38 | |
| fungi | i saw "ansible-playbook --tags build,standalone,octavia" and immediately though it could be an octavia job | 17:39 |
| fungi | s/though/thought/ | 17:39 |
| fungi | but it's just a job using octavia (one of many, i'm sure) | 17:39 |
| johnsom | Yep, they are around... lol | 17:40 |
| fungi | anyway, i did not mean to summon you, was just trying to work out what jobs we have running on coreos since we don't have any coreos nodes, that's all ;) | 17:42 |
| johnsom | No worries | 17:42 |
| clarkb | fungi: corvus we could add a cleanup task that checked for port 111 use | 17:43 |
| clarkb | then skim the zuul logs after a week | 17:43 |
| fungi | well, tons of jobs run on images with a portmapper listening, the trick is working out which ones expose it through iptables | 17:44 |
| clarkb | nc $public_ip 111 ? | 17:44 |
| fungi | does nc do udp? | 17:44 |
| clarkb | yes | 17:44 |
| fungi | neat! | 17:44 |
| corvus | clarkb, fungi, mgagne: it seems to me that the most likely explanation is that the scanning process performs the customer lookup long after it identifies a host with an open port, and we're just getting mis-routed reports. do we really need to take any further action? | 17:45 |
| clarkb | though I think you need respionses for it to be useful? | 17:45 |
| fungi | nc -u per the manpage | 17:45 |
| clarkb | corvus: mgagne says the IPs are per tenant so that IP will only ever be used by us | 17:45 |
| fungi | corvus: mgagne says we have dedicated ip addresses | 17:45 |
| mgagne | corvus: This would mean the scanner got the IP wrong. The IPs are dedicated to your account. | 17:45 |
| corvus | oh, interesting | 17:46 |
| corvus | well, it either got the ip or the time wrong | 17:46 |
| fungi | so still possible they were wrong about the timezone it's using, yes | 17:46 |
| mgagne | corvus: I asked more details about the time to the other department. | 17:46 |
| fungi | well, or somehow nodepool isn't successfully deleting nodes sometimes and isn't telling us? | 17:46 |
| corvus | it's only as good as openstack | 17:47 |
| fungi | indeed... so, flawless right? ;) | 17:47 |
| fungi | i suppose someone with access to the nova logs may also be able to tell us whether there was an instance in our project/tenant matching that ip address and time | 17:49 |
| corvus | clarkb: then i believe your idea sounds pretty good, though nc may not be installed... | 17:50 |
| mgagne | fungi: I don't think the IPs are logged in Nova. Maybe Neutron port ID which is now deleted. | 17:50 |
| clarkb | corvus: ya maybe even just have the executor open a socket? | 17:50 |
| clarkb | I'm not sure what the best implementation is there but having it run from executor to test node would catch open ports | 17:51 |
| corvus | yeah, that sounds better | 17:51 |
| corvus | and wrap it in a 'when: nodepool.provider==inap' sort of thing | 17:51 |
| corvus | or i guess we could just do 'when: nodepool.public_ipv4' | 17:51 |
| clarkb | ya catching it in another cloud might be quicker | 17:52 |
| corvus | yeah that | 17:52 |
| clarkb | sincei n theory this should happen everywhere we schedule those job(s) | 17:52 |
| Shrews | fungi: if nodepool had a persistent problem with deleting nodes, it would show up in a diminishing quota, i would think | 17:52 |
| fungi | yeah, agreed | 17:53 |
| *** michael-beaver has quit IRC | 17:55 | |
| fungi | reading rfc 1057 for some clue about how to tickle a response out of it | 17:55 |
| fungi | appendix a indicates it's a binary protocol | 17:57 |
| *** lbragstad_ has joined #openstack-infra | 17:58 | |
| fungi | part of the challenge with the netcat idea is that with udp you're not going to be able to determine the service is listening without sending something to it which elicits a response | 17:58 |
| fungi | there's no handshake to be able to say "there's something listening there which i've established a connection with" | 17:59 |
| clarkb | fungi: right we need a resposne to confirm it is there | 17:59 |
| fungi | which, my initial skimming of rfc 1057 appendix a suggests we'll need to send it an 8-bit packed structure | 18:00 |
| *** lbragstad has quit IRC | 18:00 | |
| johnsom | You might also consider rpcinfo | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| fungi | oh! right you are ;) | 18:00 |
| fungi | using an actual client which speaks the protocol may make more sense | 18:00 |
| fungi | johnsom with the obvious answers | 18:01 |
| corvus | we'll need to install that on the executors, but should be fine | 18:01 |
| fungi | yeah, would need the rpcbind package | 18:02 |
| fungi | which will additionally pull in libtirpc1 | 18:02 |
| fungi | so not especially onerous | 18:02 |
| fungi | and syntax should just be `rpcinfo W.X.Y.Z` | 18:03 |
| johnsom | I think it's rpcinfo -p <ip> | 18:03 |
| *** jaosorior has quit IRC | 18:04 | |
| fungi | -p tells it to use rpcbind v2 (portmapper) protocol | 18:04 |
| fungi | but yeah | 18:04 |
| johnsom | Ah, yeah, sorry, a bit rusty on the old portmap stuff | 18:05 |
| fungi | same, i ceased running nfs servers some years ago | 18:05 |
| *** ociuhandu has quit IRC | 18:05 | |
| fungi | i think by default modern rpcbind tries version 4 of the protocol if you don't specify | 18:05 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add rpcbind to executors https://review.opendev.org/699474 | 18:06 |
| corvus | may as well get that started | 18:07 |
| mgagne | fungi, corvus: I found timestamp of IP usage: http://paste.openstack.org/show/FZKSHVmW9iGnhL5Kzgrb/ | 18:08 |
| mgagne | based on port.create.end event in Neutron. | 18:08 |
| fungi | thanks! so that's creation time and instance uuid? | 18:11 |
| mgagne | the created_at field as found in the event for the neutron port. | 18:12 |
| mgagne | and device_id found in Neutron port, so this should be the instance UUID. | 18:12 |
| corvus | that doesn't have the 22:25 event i found? | 18:12 |
| mgagne | it seems not. I'll check again to make sure | 18:13 |
| fungi | i also don't see that uuid appearing in our nodepool launcher debug log for that timeframe | 18:13 |
| fungi | oh, wait, here we go | 18:14 |
| fungi | 2019-12-11 00:35:19,329 DEBUG nodepool.NodeLauncher: [node: 0013326094] Waiting for server 57205139-e4de-473e-9299-6052f6e558a9 for node id: 0013326094 | 18:14 |
| fungi | 2019-12-11 01:12:03,456 INFO nodepool.DeletedNodeWorker: Deleting used instance 57205139-e4de-473e-9299-6052f6e558a9 from inap-mtl01 | 18:14 |
| corvus | what's the idea here? do we want to gather a set of builds within a window around the time and try to cross that with a second report to narrow it down? | 18:15 |
| fungi | 2019-12-11 00:40:34,995 DEBUG nodepool.NodeLauncher: [node: 0013326094] Node 0013326094 is running [region: mtl01, az: nova, ip: 198.72.124.78 ipv4: 198.72.124.78, ipv6: , hostid: 198627a086ccf9500de7782a9fdb952c2599f13005bf30b2f02bcb9f] | 18:15 |
| *** rlandy has quit IRC | 18:15 | |
| mgagne | maybe trying to correlate with a job name? | 18:15 |
| fungi | ohh | 18:15 |
| fungi | wrong date | 18:16 |
| corvus | if so, what's the window size we want? are we thinking it's a EST/UTC difference? or...? | 18:16 |
| *** rlandy has joined #openstack-infra | 18:16 | |
| mgagne | I asked again the other department, they said: "The timestamp is detection time and is UTC" | 18:16 |
| fungi | i was looking at 2019-12-11 and thinking 2019-12-12, sorry | 18:16 |
| mgagne | well, maybe VM got created ~3-4 hours ago before being used and/or detected. | 18:17 |
| fungi | mgagne: so no instances created on 2019-12-12 in that timeframe? | 18:17 |
| mgagne | nothing before 6am | 18:17 |
| corvus | okay, so we have a scanning process that definitively identifies an ip+time, and a nodepool process that definitively says that ip address was not in use at that time. | 18:18 |
| fungi | yeah, the examples there end at 18:02 utc | 18:18 |
| corvus | it seems the neutron logs are backing up the nodepool logs. | 18:18 |
| mgagne | before 12:00: | 18:18 |
| fungi | we don't have a neutron log entry confirming deletion of the port for 619ae072-a88d-411a-9c12-4b0f564219f7 i guess | 18:19 |
| corvus | well, that's not the last use anyway. 3e7996a7-8e14-49f8-a13a-f0a7620bc17e is | 18:19 |
| fungi | so it's theoretically possible that instance was created at 2019-12-11T18:02:27Z and still running 6+ hours later when the scan claims to have hit it | 18:19 |
| mgagne | a240db8a-dc9f-4379-8d4e-f35fc9a1ea05 2019-12-12T07:00:48Z | 18:19 |
| mgagne | b9bc23fb-13ae-42cb-9ab9-4d3457bf5450 2019-12-12T09:40:06Z | 18:19 |
| mgagne | a14b014d-c190-4f86-9a28-491d5170de81 2019-12-12T11:39:23Z | 18:19 |
| fungi | corvus: right, and that's not included | 18:20 |
| fungi | which suggests there's still some missing data | 18:20 |
| corvus | if we want to examine the idea that a node existed for longer than nodepool thinks, then we should look at 3e7996a7-8e14-49f8-a13a-f0a7620bc17e | 18:25 |
| corvus | which is the usage immediately before the report time | 18:25 |
| corvus | ideally, mgagne will find the missing neutron logs that confirm that was the immediate prior use. but if not, we should also look at 619ae072-a88d-411a-9c12-4b0f564219f7 | 18:26 |
| corvus | i will find the build for 3e7.... | 18:26 |
| clarkb | I'll admit to having gotten a bit lost in the discussion. Let me know if there is anything I can do to help. | 18:29 |
| mgagne | 3e7996a7-8e14-49f8-a13a-f0a7620bc17e had 198.72.124.78 with port created_at 2019-12-11T22:25:47Z | 18:29 |
| mgagne | So it seems the timestamp I'm used to filter event isn't right. It's a unix timestamp. | 18:29 |
| corvus | great, that means nodepool and neutron agree about the node immediately prior | 18:29 |
| corvus | i'm still tracking down that build (it will take a minute, it appears to be an incomplete build) | 18:29 |
| corvus | also, amusingly, i have managed to open the maximum number of X windows on my workstation | 18:30 |
| clarkb | thats a thing? | 18:31 |
| corvus | apparently! i couldn't open any more windows until i closed some. maybe it's an xfce thing? | 18:32 |
| clarkb | TIL | 18:32 |
| *** jpena is now known as jpena|off | 18:33 | |
| corvus | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3b/698145/4/gate/cross-osc-tox-docs/c3b9dba/ | 18:33 |
| corvus | that job seems particularly unlikely to do anything weird with port 111 | 18:33 |
| mgagne | corvus: somehow the port.create.end event is missing for the 3e7996a7-8e14-49f8-a13a-f0a7620bc17e instance. That's why I missed it. | 18:34 |
| corvus | mgagne: could that indicate some kind of error in neutron? perhaps one where the ip is assigned to something without correct auditing? | 18:35 |
| mgagne | the auditing is done by stacktach. that's where I'm looking at. | 18:36 |
| mgagne | I'll grep for something else, the IP looks to be in the compute.instance.create.end payload too. | 18:36 |
| corvus | on the inap mirror, i see no requests from 198.72.124.78 on 2019-12-12 before 06:04:25 | 18:38 |
| *** pcaruana has quit IRC | 18:38 | |
| mgagne | I think I have a better query now: http://paste.openstack.org/show/P8WtmyvKpdaABlXCXOgr/ | 18:40 |
| corvus | and the last request before that is 22:32:06 | 18:40 |
| corvus | which does correspond with the pip install at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3b/698145/4/gate/cross-osc-tox-docs/c3b9dba/job-output.txt | 18:41 |
| mgagne | so last one is 3e7996a7-8e14-49f8-a13a-f0a7620bc17e | 18:41 |
| mgagne | which got deleted at "2019-12-11T22:35:29.000000". So this makes no sense. | 18:42 |
| corvus | so nodepool and neutron agree that 3e7996a7-8e14-49f8-a13a-f0a7620bc17e is the most recent instance with that ip before the reporting time. the mirror and build logs confirm that the expected node did have that ip and was using it during the time that nodepool expected that node to have the ip. nodepool says nothing had it for 6 hours after that (the report time is during that window). and the | 18:43 |
| corvus | mirror confirms that nothing was accessing it from that ip during that window (so that's a confirmation external to nodepool that no rogue job had started after 3e7996a7-8e14-49f8-a13a-f0a7620bc17e) | 18:43 |
| fungi | cool, this sounds like actionable feedback for the security department in that case | 18:44 |
| fungi | thanks for investigating! | 18:45 |
| *** panda has quit IRC | 18:49 | |
| *** panda has joined #openstack-infra | 18:49 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Fix ansible use of filters and tests https://review.opendev.org/699478 | 18:50 |
| *** jamesmcarthur has quit IRC | 18:51 | |
| mgagne | corvus, fungi: I checked and there is no rogue instance with that IP address. Only one is a legit instance referenced in Nova. | 18:54 |
| *** dave-mccowan has joined #openstack-infra | 18:54 | |
| fungi | yeah, the mystery deepens | 18:54 |
| fungi | starting to seem increasingly likely they reported the wrong address or wrong timestamp | 18:55 |
| fungi | (or that the timestamp is uselessly inaccurate in some way at least) | 18:55 |
| fungi | here is the exact text of what they sent us: http://paste.openstack.org/show/787687/ | 18:57 |
| fungi | er, sorry, http://paste.openstack.org/show/787688/ is slightly cleaner, the previous one had a linewrap added by my mua | 18:58 |
| *** jaosorior has joined #openstack-infra | 19:00 | |
| *** ralonsoh has quit IRC | 19:04 | |
| *** pcaruana has joined #openstack-infra | 19:18 | |
| *** hashar has joined #openstack-infra | 19:31 | |
| *** eharney has quit IRC | 19:45 | |
| *** smarcet has joined #openstack-infra | 19:45 | |
| *** ianw is now known as ianw_pto | 19:50 | |
| ianw_pto | fyi i'm on PTO until Jan 13 now ... never that far from a computer but won't be actively pushing on stuff | 19:51 |
| fungi | i'll be away from the computer starting tomorrow, until new year's eve | 19:52 |
| ianw_pto | sounds fun! | 19:53 |
| fungi | that's the general idea ;) | 19:54 |
| *** jamesmcarthur has joined #openstack-infra | 19:55 | |
| *** pcaruana has quit IRC | 19:59 | |
| AJaeger | fungi, ianw_pto, enjoy your vacations! | 20:10 |
| *** jamesmcarthur has quit IRC | 20:11 | |
| *** jamesmcarthur has joined #openstack-infra | 20:12 | |
| fungi | thanks! | 20:16 |
| clarkb | before everyone disappears we should fix the mysql backups | 20:18 |
| clarkb | fungi: were you working on that or should someone else take a look? | 20:18 |
| mordred | clarkb: from meeting scrollback - yes, just bumping the gitea tag number in the dockerfile should be all that's needed | 20:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Bump gitea version to 1.10.1 https://review.opendev.org/699490 | 20:19 |
| mordred | clarkb: ^^ there ya go | 20:19 |
| clarkb | mordred: no template delta? | 20:19 |
| *** jamesmcarthur has quit IRC | 20:19 | |
| clarkb | I'm going to eat lunch then start looking at reviews | 20:19 |
| mordred | clarkb: will go check | 20:19 |
| mordred | clarkb: https://review.opendev.org/#/c/699406 is needed for the gerrit stack | 20:20 |
| mordred | (and the one right after it) | 20:20 |
| mordred | https://review.opendev.org/#/c/690511 | 20:20 |
| mordred | the rest of the stack is reviewed and green | 20:21 |
| mordred | (I'll keep poking at getting that in and moving forward over the slow period) | 20:21 |
| clarkb | both of those changes lgtm | 20:21 |
| mordred | clarkb: also - no template changes in 1.10.1 | 20:24 |
| *** Lucas_Gray has joined #openstack-infra | 20:25 | |
| clarkb | cool | 20:25 |
| clarkb | now that is really curious | 20:27 |
| clarkb | mysqlclient on etherpad is working find | 20:27 |
| clarkb | *fine | 20:27 |
| clarkb | it has no ssl config but does not require it | 20:27 |
| clarkb | mysql --version reports it has the same mysqlclient version as review.o.o | 20:27 |
| clarkb | mordred: ^ can you help decipher that and the way to fix db backups on review.o.o? | 20:27 |
| mordred | uhm. sure! | 20:28 |
| mordred | where are we seeing issues with ssl? | 20:28 |
| clarkb | mordred: on review.o.o if you try to connect to the gerrit db using mysql client and the root user my.cnf it fails on ssl. iF you run mysql --skip-ssl it works | 20:29 |
| clarkb | mordred: doing the same on etherpad.o.o works fine without the --skip-ssl and it has the same mysqlclient version | 20:29 |
| clarkb | I'm guessing there is some piece of config somewhere we hve on review.o.o and not on etherpad or vice versa | 20:29 |
| clarkb | /etc/mysql/conf.d/client.conf exists on review.o.o but not etherpad. However it doesn't seem to say anything about ssl on review.o.o | 20:29 |
| mordred | I don't see any ssl config on review.o.o | 20:30 |
| mordred | yeah | 20:30 |
| mordred | that said: | 20:30 |
| mordred | ssl TRUE | 20:30 |
| clarkb | where do you see that? | 20:30 |
| mordred | looking more | 20:30 |
| clarkb | kk | 20:30 |
| mordred | mysql --help | 20:30 |
| clarkb | thanks | 20:30 |
| clarkb | I'm going to eat and can help dig more after | 20:30 |
| mordred | clarkb: etherpad and review seem identical | 20:33 |
| mordred | both show SSL TRUE with --no-defaults - so I think that's a red herring | 20:34 |
| mordred | maybe the db for reviewdb is advertising ssl for some reason | 20:34 |
| *** jamesmcarthur has joined #openstack-infra | 20:34 | |
| mordred | and since ssl is true, the client is trying to use it - but there's no certs so it can't? | 20:34 |
| clarkb | oh server + client setup affecting it? | 20:35 |
| clarkb | I guess if we set ssl off in yhe cnf that shoulf make it just work? | 20:35 |
| mordred | lemme try | 20:35 |
| mordred | yes | 20:35 |
| clarkb | ok so update puppet-mysql_backup with that is our fix I think | 20:36 |
| mordred | clarkb: [client]\nssl=false fixes it | 20:36 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-mysql_backup master: Turn off ssl in my.cnf https://review.opendev.org/699494 | 20:37 |
| mordred | clarkb: ^^ | 20:37 |
| fungi | clarkb: i doubt i have time to dig deeper into the mysql behavior change on review.o.o, but adding --skip-ssl to mysqldump in our cronjob should be sufficient? | 20:38 |
| fungi | oh, or what mordred suggested now that i'm caught up on scrollback | 20:40 |
| clarkb | fungi: I think the my.cnf is better do that all client commands work | 20:40 |
| fungi | thanks! | 20:40 |
| *** pcaruana has joined #openstack-infra | 20:42 | |
| *** eharney has joined #openstack-infra | 20:42 | |
| *** jaosorior has quit IRC | 20:45 | |
| *** gfidente has quit IRC | 20:48 | |
| openstackgerrit | Merged zuul/nodepool master: Dockerfile: install nodepool-builder dependencies https://review.opendev.org/693306 | 20:50 |
| *** smarcet has quit IRC | 20:54 | |
| *** pkopec has quit IRC | 20:55 | |
| openstackgerrit | Merged zuul/nodepool master: Add a container-with-releases functional test https://review.opendev.org/698818 | 20:58 |
| *** armax has joined #openstack-infra | 20:59 | |
| *** kozhukalov has joined #openstack-infra | 21:02 | |
| mordred | corvus, fungi: have a sec for 2 quick patches? https://review.opendev.org/#/c/699406 and https://review.opendev.org/#/c/690511/ would be nice to land | 21:02 |
| corvus | done | 21:03 |
| *** kopecmartin is now known as kopecmartin|off | 21:04 | |
| *** smarcet has joined #openstack-infra | 21:05 | |
| *** jaosorior has joined #openstack-infra | 21:09 | |
| mgagne | corvus, fungi: so I checked with abuse/security department. Actual time is 2019-12-11 18:43:56 UTC. They will review the time conversion mapping settings. | 21:09 |
| corvus | mgagne: progress! i'll look that up | 21:10 |
| mgagne | corvus, fungi: So this would match a1f4fbbc-4d8a-4f17-8848-53e15da23819 @ 2019-12-11 18:28:17+00:00 | 21:10 |
| corvus | mgagne: that does agree with nodepool; looking up the build now | 21:12 |
| mordred | corvus: thanks! | 21:12 |
| corvus | here's the build: http://zuul.opendev.org/t/openstack/build/3d1da15f262f47c0b1d15a89904dc849 | 21:13 |
| corvus | ironic-tempest-ipa-partition-pxe_ipmitool-tinyipa | 21:13 |
| mriedem | clarkb: donnyd: another grenade job failed networking with fortnebula https://zuul.opendev.org/t/openstack/build/79107b8c3bac4fdba4d21a059311e9c3/log/logs/devstack-gate-setup-host.txt#3408 | 21:13 |
| corvus | clarkb, fungi: ^^ see convo with mgagne | 21:13 |
| *** smarcet has quit IRC | 21:14 | |
| *** jaosorior has quit IRC | 21:15 | |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%7C%20localhost%20%7C%20FAILED%20%7C%20network_sanity_check%20%3A%20Perform%20ping%20check%20%7C%20rc%3D1%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d - far and away FN for those failures | 21:15 |
| mriedem | looks like mostly jobs that use devstack-gate, i don't know if there is something about how it does network setup checking that is different in the zuulv3 jobs | 21:16 |
| fungi | corvus: mgagne: thanks! in that case we probably need to forget the analysis i performed a week ago, as it was based on incorrect time assumptions and so likely selected an unrelated build (though maybe we still have logs from then and could redo the analysis?) | 21:18 |
| fungi | i need to go meet some folks for dinner, but will hopefully be back in an hour or so | 21:18 |
| mgagne | fungi: abuse/security told me that a misconfiguration made it so now() was used instead of original report timestamp. | 21:18 |
| *** smarcet has joined #openstack-infra | 21:18 | |
| clarkb | mriedem: we know we had a network issue to the mirror early monday iirc | 21:20 |
| clarkb | however that timestamp is today | 21:20 |
| mriedem | yeah a spike in this failure since yesterday | 21:20 |
| clarkb | the host is reachable via ipv4 externally | 21:20 |
| clarkb | donnyd: ^ any chance there are still ipv6 route issues? | 21:20 |
| donnyd | possible | 21:20 |
| clarkb | fungi: corvus: and we need to work with ironic to figure out why their job is exposing 111? | 21:21 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for network_sanity_check ping check fail bug 1856760 https://review.opendev.org/699503 | 21:21 |
| openstack | bug 1856760 in OpenStack-Gate "icmp_seq=1 Destination unreachable: Address unreachable causing built failures on fortnebula nodes" [Undecided,New] https://launchpad.net/bugs/1856760 | 21:21 |
| mriedem | there are a handful of rax and 1 inap hit on that but 40 on FN | 21:21 |
| clarkb | https://f9c248a400bd30174240-0e9efd411d5f516ecd1b5a61c03e35b7.ssl.cf1.rackcdn.com/697585/6/check/ironic-tempest-ipa-partition-pxe_ipmitool-tinyipa/3d1da15/controller/logs/iptables.txt.gz is what iptables looks like at the end of the job I think | 21:22 |
| donnyd | maybe FN is just so awesome that I get all the jobs ;-) | 21:22 |
| donnyd | LMAO | 21:22 |
| donnyd | I reset the GW | 21:23 |
| donnyd | that was what i had to do to fix the CI project | 21:23 |
| donnyd | mriedem: can you recheck please | 21:23 |
| clarkb | that iptables ruleset doesn't appear to allow external udp to port 111 | 21:24 |
| mriedem | donnyd: done | 21:24 |
| clarkb | I see the host dropping other udp packets in syslog (different dpt) | 21:26 |
| clarkb | that implies the firewall is generally working | 21:27 |
| *** jamesmcarthur has quit IRC | 21:28 | |
| donnyd | i can reach all of the instances inbound | 21:28 |
| *** kozhukalov has quit IRC | 21:29 | |
| mgagne | fungi, corvus: time was AM, not PM | 21:30 |
| mgagne | 2019-12-11 06:43:56 | 21:30 |
| clarkb | ok so different job? that would make sense as the identified job seems clean | 21:30 |
| mgagne | a59f143c-9006-4e92-8c05-e82f4864f03a @ 2019-12-11 06:00:45+00:00 | 21:30 |
| corvus | ok, i'll look that one up | 21:31 |
| corvus | that also agrees with nodepool; node existed from 6:00 to 8:18 | 21:32 |
| corvus | http://zuul.opendev.org/t/openstack/build/6fc5285fdb76484b894f0d288facdbb2 | 21:35 |
| corvus | openstack-helm-multinode-temp-ubuntu | 21:35 |
| corvus | that's a 5-node job; the ip in question is the "primary" host | 21:35 |
| clarkb | I wonder if they use our "turn off the firewall" role for k8s | 21:35 |
| *** Lucas_Gray has quit IRC | 21:37 | |
| corvus | looks like they have their own | 21:37 |
| corvus | the result is this: http://zuul.opendev.org/t/openstack/build/6fc5285fdb76484b894f0d288facdbb2/console | 21:37 |
| corvus | er | 21:37 |
| corvus | http://zuul.opendev.org/t/openstack/build/6fc5285fdb76484b894f0d288facdbb2/console#2/1/7/primary | 21:37 |
| clarkb | that does look like an empty ruleset | 21:38 |
| clarkb | http://zuul.opendev.org/t/openstack/build/6fc5285fdb76484b894f0d288facdbb2/console#2/1/6/primary does the rule update I think | 21:39 |
| clarkb | now I guess we have to decide if disabling the firewall is a valid tactic for dealing with k8s | 21:40 |
| *** pcaruana has quit IRC | 21:41 | |
| clarkb | does k8s do anything with low ports? | 21:42 |
| clarkb | dns maybe? | 21:42 |
| clarkb | possible that we can open 1024 and above | 21:42 |
| clarkb | but leave the low ports firewalled off | 21:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix ansible use of filters and tests https://review.opendev.org/699478 | 21:44 |
| clarkb | we could resurrect the idea of relying on security groups | 21:45 |
| clarkb | that is probably going to be most zuul user friendly | 21:46 |
| clarkb | but will need careful application | 21:46 |
| *** rcernin has quit IRC | 21:46 | |
| corvus | or modify that job to add a reject rule for 111 | 21:46 |
| clarkb | corvus: ya but its going to be a similar issue with dns resolvers and other services that can be exploited for reflection attacks | 21:47 |
| clarkb | trying to do it port by port seems like we will always be behind where we want to be | 21:47 |
| clarkb | I guess its largely just udp services though | 21:49 |
| clarkb | maybe we can accept tcp? | 21:49 |
| clarkb | and block udp | 21:49 |
| *** smarcet has joined #openstack-infra | 21:51 | |
| openstackgerrit | Merged opendev/system-config master: Update bazel to version 1.2.0 https://review.opendev.org/699406 | 21:52 |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for network_sanity_check ping check fail bug 1856760 https://review.opendev.org/699503 | 21:52 |
| openstack | bug 1856760 in OpenStack-Gate "icmp_seq=1 Destination unreachable: Address unreachable causing built failures on fortnebula nodes" [Undecided,New] https://launchpad.net/bugs/1856760 | 21:52 |
| clarkb | mnaser: ^ you may have thoughts on that since you are doing a fair but of zuul + k8s intersection work too iirc | 21:52 |
| * mnaser reads | 21:52 | |
| *** dpawlik has quit IRC | 21:53 | |
| *** ijw has joined #openstack-infra | 21:54 | |
| *** stevebaker_ is now known as stevebaker | 21:54 | |
| mnaser | I don’t know why they aren’t blocking that port, there’s no need to unblock it | 21:54 |
| mnaser | The cluster DNS generally runs in the overlay network so that should affect things | 21:54 |
| clarkb | mnaser: well they are unblocking everything. And I think corvus decided this was a reasonable action for zuul's k8s jobs too | 21:55 |
| *** rascasoft has joined #openstack-infra | 21:55 | |
| clarkb | mnaser: hrm if things run in overlays I guess its just the cluster networking and not the hosted service networking we'd need to worry about? | 21:55 |
| clarkb | api access, the overlays themselves, etc | 21:55 |
| mnaser | I guess depending on the overlay that you’re using all you might realistically need to open are high number ports like 30k and above if you’ll be exposing NodePort services | 21:56 |
| mnaser | And obviously technically Kubernetes can let you run containers that use the host networking so that would entirely skip the overlay | 21:56 |
| clarkb | our multinode networking does open all ports between nodeset members though so we must be missing something | 21:57 |
| mnaser | What overlay is being used? I’m on mobile so I can dig deeper right now | 21:57 |
| corvus | clarkb: that is not a "multinode" job | 21:57 |
| corvus | it's just a job with multiple nodes | 21:57 |
| clarkb | corvus: ah | 21:57 |
| clarkb | mnaser: re overlay I have no idea its an osh job | 21:57 |
| clarkb | mnaser: for zuul its whatever minikube deploys by default | 21:57 |
| *** diablo_rojo has quit IRC | 21:58 | |
| mnaser | ah well my guess is “it’s the easiest way to avoid figuring out why it ain’t working” was the case for osh | 21:58 |
| corvus | (ie, it does not inherit from the job "multinode") | 21:58 |
| clarkb | so ya they probably couldn't get nodeset nodes to talk to each other to make a cluster. Then disabled the firewall and now it works :/ | 21:59 |
| clarkb | though there likely is additional firewall tweaking necessary on top of what multinode firewall setup would give you | 22:00 |
| *** ijw has quit IRC | 22:04 | |
| *** panda has quit IRC | 22:07 | |
| *** panda has joined #openstack-infra | 22:10 | |
| clarkb | tosky: left a couple notes on https://review.opendev.org/#/c/548936 I don't think anything is a hard -1 though the hosts: all may need to be updated if things are reliable that way | 22:11 |
| clarkb | tosky: for the grenade.sh vs devstack.sh I think I'm mostly looking for a bit more of the reason why that change is made. | 22:11 |
| clarkb | corvus: mnaser fungi mgagne maybe the thing to do is start a thread on openstack-discuss about this issue and see if people that know k8s can suggest an appropriate solution? | 22:13 |
| tosky | clarkb: uhm, I'm not sure I get the second comment; is that about letting devstack roles do the devstack part? | 22:15 |
| tosky | run-grenade only runs grenade.sh, excluding the installation part which is already done, and it should do it as it was done before | 22:16 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-mysql_backup master: Turn off ssl in my.cnf https://review.opendev.org/699494 | 22:17 |
| clarkb | mordred: fungi corvus ^ that fixes the puppet linter errors | 22:17 |
| tosky | the point of splitting was already there since the first change done by Andrea, and I guess it was to reduce the duplication | 22:17 |
| tosky | the ansible roles in devstack.git already know how to deploy | 22:17 |
| clarkb | tosky: well in the old d-g driver it runs grenade.sh and it does the installation and the upgrade | 22:17 |
| clarkb | tosky: this change runs stack.sh for the installation then grenade.sh only for the upgrade | 22:18 |
| tosky | yes, exactly | 22:18 |
| clarkb | and ya I guess that is probably why? we arleady have a devstack role for the upgrade? | 22:18 |
| clarkb | er installation | 22:18 |
| tosky | that was my understanding when I continue Andrea's work | 22:18 |
| clarkb | got it | 22:18 |
| clarkb | corvus: if you think that would be a good next step for the port 111 thing I can write that email to openstack-discuss | 22:19 |
| tosky | as there will be other reviews, I will add some notes | 22:20 |
| tosky | what is the record for the review with the highest amount of changesets? :) | 22:20 |
| corvus | clarkb: ++ | 22:20 |
| *** rh-jelabarre has quit IRC | 22:26 | |
| *** dave-mccowan has quit IRC | 22:30 | |
| *** diablo_rojo has joined #openstack-infra | 22:35 | |
| *** slaweq has quit IRC | 22:37 | |
| sshnaidm | I have an error in one of jobs, FYI: ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='opendev.org', port=443): Max retries exceeded with url: /openstack/requirements/raw/branch/master/upper-constraints.txt (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f3a8e21c110>: Failed to establish a new connection: [Errno 101] Network is unreachable',)) | 22:40 |
| sshnaidm | https://43a0ea1d907a03afa010-dc7530dd32f75128221ba69b2574d743.ssl.cf5.rackcdn.com/699314/1/check/tripleo-ci-centos-7-scenario001-standalone/5c69614/job-output.txt | 22:40 |
| *** rcernin has joined #openstack-infra | 22:42 | |
| clarkb | sshnaidm: any idea why it is trying to reach a remote resource in the first place? | 22:43 |
| clarkb | the command shows it is using a local constraints file | 22:43 |
| sshnaidm | clarkb, no idea.. maybe because of -U ? | 22:44 |
| clarkb | packages can't provide their own constraints though, they have to be provided directly to the command. This is odd | 22:45 |
| clarkb | sshnaidm: that said network is unreachable implies to me a layer 3 or below problem | 22:46 |
| clarkb | like you don't have a route to the opendev.org ip addresses | 22:46 |
| sshnaidm | I hope it's just a sort of glitch, will keep eye on it | 22:48 |
| clarkb | sshnaidm: do you capture any networking info for the host at the end of the job | 22:49 |
| sshnaidm | clarkb, oh, yeah :) https://43a0ea1d907a03afa010-dc7530dd32f75128221ba69b2574d743.ssl.cf5.rackcdn.com/699314/1/check/tripleo-ci-centos-7-scenario001-standalone/5c69614/logs/undercloud/var/log/extra/network.txt.gz | 22:50 |
| sshnaidm | https://43a0ea1d907a03afa010-dc7530dd32f75128221ba69b2574d743.ssl.cf5.rackcdn.com/699314/1/check/tripleo-ci-centos-7-scenario001-standalone/5c69614/logs/undercloud/var/log/extra/netstat.txt.gz | 22:50 |
| clarkb | sshnaidm: https://43a0ea1d907a03afa010-dc7530dd32f75128221ba69b2574d743.ssl.cf5.rackcdn.com/699314/1/check/tripleo-ci-centos-7-scenario001-standalone/5c69614/zuul-info/zuul-info.primary.txt is the data from the beginning of the job | 22:50 |
| sshnaidm | clarkb, in the beginning all was good, traceroute to opendev.org worked | 22:51 |
| clarkb | yup | 22:52 |
| clarkb | and the routes at the end look similar (just additions for the br-ex network) | 22:52 |
| clarkb | I wonder if it tried to do ipv6 for some reason | 22:52 |
| clarkb | or maybe the firewall dropped the packets | 22:52 |
| openstackgerrit | Merged opendev/system-config master: Use explicit image paths https://review.opendev.org/690511 | 22:52 |
| clarkb | syslog doesn't log any dropped port 443 packets | 22:54 |
| clarkb | I know in the past there have been dns issues that looked like other problems | 22:54 |
| clarkb | I wonder if this could actually be a name resolution problem? | 22:54 |
| clarkb | mostly it looks like it isn't even really trying to make the tcp connection. its just deciding very quickly it can't do it for some reason | 22:55 |
| clarkb | rather than sending a bunch of SYNs looking for a friend | 22:55 |
| clarkb | but there was no ACK to find | 22:55 |
| sshnaidm | we have unbound log: https://43a0ea1d907a03afa010-dc7530dd32f75128221ba69b2574d743.ssl.cf5.rackcdn.com/699314/1/check/tripleo-ci-centos-7-scenario001-standalone/5c69614/logs/undercloud/var/lib/unbound/unbound.log.txt.gz | 23:00 |
| sshnaidm | seems like it resolves | 23:01 |
| sshnaidm | hmm.. but AAAA | 23:01 |
| sshnaidm | and with A: 38.108.68.124 | 23:02 |
| *** armax has quit IRC | 23:07 | |
| *** dklyle_ is now known as dklyle | 23:12 | |
| *** tkajinam has joined #openstack-infra | 23:13 | |
| openstackgerrit | Clark Boylan proposed openstack/devstack-gate master: Bump ANSIBLE_VERSION default to 2.7.14 https://review.opendev.org/699463 | 23:15 |
| clarkb | mriedem: ^ I think that edit is needed for the linter job | 23:15 |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 23:16 |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 23:17 |
| clarkb | sshnaidm: thinking a bit more we might want to sort out where the remote constraints file comes from as that may give us a hint to why it is failing | 23:18 |
| clarkb | basically pip is acting in an unexpected manner and working backward from the known unexpected thing may help | 23:18 |
| sshnaidm | yeah, possibly | 23:19 |
| *** sgw has quit IRC | 23:19 | |
| *** hashar has quit IRC | 23:20 | |
| clarkb | https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_install.yml#L100-L121 this is the task that is failing right? | 23:21 |
| clarkb | I wonder if you can have transitive constraints and that is how this happens | 23:22 |
| clarkb | one of those local files lists the remote | 23:22 |
| clarkb | yes that is what happens | 23:24 |
| clarkb | via https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/utility-install.yml#L25 and https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/utility-install.yml#L126-L127 | 23:25 |
| clarkb | thats not totally confusing at all | 23:25 |
| clarkb | also in a testing environment its wrong to go to the network for that when the repo is available locally | 23:25 |
| openstackgerrit | Merged opendev/puppet-mysql_backup master: Turn off ssl in my.cnf https://review.opendev.org/699494 | 23:25 |
| clarkb | but at least we undersatnd that bit now | 23:25 |
| fungi | and i guess nothing is passing a requirements_git_url value in to that playbook | 23:27 |
| *** dchen has joined #openstack-infra | 23:28 | |
| clarkb | well and it should just be a file path (though I guess you can express that as a url too) | 23:28 |
| fungi | right, it could just be the path to the file in the checked-out copy of openstack/requirements | 23:29 |
| fungi | which would also make it possible to successfully depends-on proposed changes to the constraints file too | 23:29 |
| clarkb | gitea03 seems sad, but its universally sad and not just against that single url | 23:30 |
| clarkb | it was just restarted | 23:30 |
| clarkb | looks like due to a mariadb update | 23:30 |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| clarkb | that file loads fine from it now and all of the others | 23:31 |
| clarkb | (the speed at which it fails really makes me suspect local networking though) | 23:31 |
| *** ociuhandu has quit IRC | 23:35 | |
| *** smarcet has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!