| clarkb | ianw: ^ off the top of my head that cleanup can be done | 00:00 |
|---|---|---|
| clarkb | I think all the fuel entries can be removed from project-config | 00:00 |
| clarkb | AJaeger: ^ can you confirm | 00:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix python3 compat in tox siblings handling https://review.opendev.org/698335 | 00:00 |
| ianw | so it still doesn't boot with a xenial-built-initramfs (i copied the initramfs from a nb .qcow2) ... i wonder if bionic era mkfs is somehow making a disk that fools the trusty era image that it is unclean | 00:02 |
| *** aaronsheffield has quit IRC | 00:05 | |
| ianw | yes ... ! "/dev/vda1 has unsupported features: metadata_csum" | 00:08 |
| *** rf0lc0 has quit IRC | 00:10 | |
| clarkb | looks like we can tune2fs that off | 00:13 |
| clarkb | I wonder if that is sufficient to make it happy | 00:13 |
| clarkb | linux 3.6 is where it landed | 00:13 |
| *** igordc has quit IRC | 00:13 | |
| clarkb | oh we can just mkfs.ext4 -O^metadata_csum | 00:14 |
| ianw | hopefully ... | 00:14 |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| ianw | i wonder why the "rw" lets it mount ... i guess that bypasses the fsck | 00:22 |
| *** rkukura has quit IRC | 00:24 | |
| ianw | ok, it looks like hacking in ^metadata_csum lets it boot | 00:28 |
| fungi | clarkb: we test nothing of wiki on trusty afaik. the only thing with a modicum of puppeting is wiki-dev which is xenial | 00:30 |
| fungi | the production wiki is not puppet-managed | 00:30 |
| openstackgerrit | Merged zuul/zuul master: Add upgrade note about ansible_python_interpreter https://review.opendev.org/698318 | 00:31 |
| fungi | and it won't get puppeted on trusty, if it survives, it'll move to xenial as soon as the puppetry is in a working state | 00:33 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: DNM test all the things on ansible 2.8 with python3 https://review.opendev.org/698344 | 00:34 |
| *** jamesmcarthur has quit IRC | 00:39 | |
| *** rkukura has joined #openstack-infra | 00:44 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 00:54 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Work around Trusty ext4 metadata_csum errors on Bionic https://review.opendev.org/698346 | 00:54 |
| *** gyee has quit IRC | 01:09 | |
| *** hwoarang has quit IRC | 01:13 | |
| *** hwoarang has joined #openstack-infra | 01:14 | |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 01:26 |
| *** goldyfruit_ has joined #openstack-infra | 01:27 | |
| *** rlandy|bbl has quit IRC | 01:49 | |
| *** larainema has joined #openstack-infra | 01:49 | |
| *** rcernin has quit IRC | 01:53 | |
| *** dave-mccowan has joined #openstack-infra | 02:00 | |
| *** goldyfruit_ has quit IRC | 02:15 | |
| *** goldyfruit_ has joined #openstack-infra | 02:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 02:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Work around Trusty ext4 metadata_csum errors on Bionic https://review.opendev.org/698346 | 02:36 |
| *** goldyfruit_ has quit IRC | 02:40 | |
| *** ricolin has joined #openstack-infra | 02:40 | |
| *** rcernin has joined #openstack-infra | 02:54 | |
| *** dave-mccowan has quit IRC | 02:55 | |
| *** jklare has quit IRC | 02:58 | |
| *** jklare has joined #openstack-infra | 02:59 | |
| *** ykarel|away has joined #openstack-infra | 03:02 | |
| *** rascasoft has quit IRC | 03:06 | |
| *** rascasoft has joined #openstack-infra | 03:09 | |
| *** factor has joined #openstack-infra | 03:31 | |
| *** psachin has joined #openstack-infra | 03:40 | |
| *** ijw_ has quit IRC | 03:49 | |
| *** ociuhandu has joined #openstack-infra | 04:04 | |
| *** ociuhandu has quit IRC | 04:08 | |
| *** ramishra has joined #openstack-infra | 04:12 | |
| *** udesale has joined #openstack-infra | 04:13 | |
| *** ykarel|away has quit IRC | 04:24 | |
| *** diablo_rojo has quit IRC | 04:29 | |
| *** ykarel|away has joined #openstack-infra | 04:41 | |
| *** rishabhhpe has joined #openstack-infra | 04:43 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 05:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Work around Trusty ext4 metadata_csum errors on Bionic https://review.opendev.org/698346 | 05:02 |
| *** dansmith has quit IRC | 05:10 | |
| *** dansmith has joined #openstack-infra | 05:14 | |
| *** surpatil has joined #openstack-infra | 05:22 | |
| *** ianychoi has joined #openstack-infra | 05:32 | |
| *** jklare has quit IRC | 05:59 | |
| *** soniya29 has joined #openstack-infra | 06:02 | |
| *** jklare has joined #openstack-infra | 06:02 | |
| *** ykarel|away is now known as ykarel | 06:03 | |
| *** pcaruana has joined #openstack-infra | 06:09 | |
| *** jtomasek has joined #openstack-infra | 06:09 | |
| *** raukadah is now known as chkumar|ruck | 06:10 | |
| *** abhishekk has joined #openstack-infra | 06:45 | |
| abhishekk | hi team, is something wrong with neutron-grenade job | 06:46 |
| abhishekk | it is failing since Monday | 06:46 |
| abhishekk | https://130a90bca998bf7bfd1f-51174786df661fdc707186ccf04b5df9.ssl.cf5.rackcdn.com/686073/3/check/neutron-grenade/9c9e738/ | 06:46 |
| abhishekk | tempest and glance projects are affected AFAIK | 06:46 |
| *** jklare has quit IRC | 06:58 | |
| *** dpawlik has joined #openstack-infra | 07:00 | |
| *** jklare has joined #openstack-infra | 07:00 | |
| AJaeger | clarkb: regarding fuel: I asked per email and there was some use of the repos for some more time - but I don't see it. I think we should remove the trusty jobs everywhere... | 07:01 |
| kevinz | ianw: there? | 07:02 |
| *** dpawlik has quit IRC | 07:04 | |
| *** dpawlik has joined #openstack-infra | 07:07 | |
| *** ianychoi_ has joined #openstack-infra | 07:08 | |
| *** tosky has joined #openstack-infra | 07:09 | |
| *** rishabhhpe has quit IRC | 07:09 | |
| *** ianychoi has quit IRC | 07:11 | |
| *** lpetrut has joined #openstack-infra | 07:18 | |
| *** witek has joined #openstack-infra | 07:23 | |
| *** ykarel is now known as ykarel|lunch | 07:30 | |
| ianw | kevinz: for a little | 07:37 |
| kevinz | iaw: good afternoon, I just have a small questions, does zuul support recheck a special jobs? | 07:38 |
| ianw | kevinz: do you mean can you run recheck against just one job? | 07:39 |
| kevinz | ianw: yes, exactly | 07:39 |
| ianw | no you can't; what i usually do is comment out the jobs i don't want to run in the .zuul file and mark it [wip] | 07:40 |
| kevinz | ianw: ack, thanks | 07:45 |
| *** slaweq has joined #openstack-infra | 07:47 | |
| *** hashar has joined #openstack-infra | 07:51 | |
| *** soniya29 has quit IRC | 07:54 | |
| *** tesseract has joined #openstack-infra | 07:57 | |
| *** jklare has quit IRC | 07:58 | |
| *** jklare has joined #openstack-infra | 08:02 | |
| *** ociuhandu has joined #openstack-infra | 08:05 | |
| *** ociuhandu has quit IRC | 08:05 | |
| *** ociuhandu has joined #openstack-infra | 08:07 | |
| *** tkajinam has quit IRC | 08:12 | |
| *** ccamacho has joined #openstack-infra | 08:13 | |
| *** pcaruana has quit IRC | 08:22 | |
| *** dchen has quit IRC | 08:23 | |
| *** ociuhandu has quit IRC | 08:30 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| *** rpittau|afk is now known as rpittau | 08:36 | |
| *** kopecmartin|off is now known as kopecmartin | 08:44 | |
| *** pkopec has joined #openstack-infra | 08:47 | |
| *** ccamacho is now known as ccamacho|pto | 08:50 | |
| *** kjackal has joined #openstack-infra | 08:50 | |
| *** jpena|off is now known as jpena | 08:53 | |
| *** ociuhandu has joined #openstack-infra | 08:54 | |
| *** ralonsoh has joined #openstack-infra | 08:55 | |
| *** strigazi has joined #openstack-infra | 08:55 | |
| *** jklare has quit IRC | 08:58 | |
| *** ociuhandu has quit IRC | 08:59 | |
| *** ociuhandu has joined #openstack-infra | 09:00 | |
| *** jklare has joined #openstack-infra | 09:01 | |
| *** iurygregory has joined #openstack-infra | 09:02 | |
| *** soniya29 has joined #openstack-infra | 09:03 | |
| *** ociuhandu has quit IRC | 09:05 | |
| *** lucasagomes has joined #openstack-infra | 09:09 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: create APP_DIR https://review.opendev.org/693646 | 09:20 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: install sudo for nodepool-builder https://review.opendev.org/694709 | 09:20 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add DEBUG environment flag https://review.opendev.org/694845 | 09:20 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 09:20 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 09:20 |
| *** ociuhandu has joined #openstack-infra | 09:27 | |
| *** udesale has quit IRC | 09:29 | |
| *** udesale has joined #openstack-infra | 09:32 | |
| *** rishabhhpe has joined #openstack-infra | 09:32 | |
| *** dpawlik has quit IRC | 09:38 | |
| *** ccamacho|pto has quit IRC | 09:38 | |
| *** ccamacho has joined #openstack-infra | 09:38 | |
| *** derekh has joined #openstack-infra | 09:43 | |
| *** factor has quit IRC | 09:46 | |
| *** factor has joined #openstack-infra | 09:47 | |
| *** apetrich has joined #openstack-infra | 09:50 | |
| *** dchen has joined #openstack-infra | 09:50 | |
| *** rcernin has quit IRC | 09:50 | |
| *** farhanjamil has joined #openstack-infra | 09:53 | |
| *** ccamacho is now known as ccamacho|pto | 09:54 | |
| *** dpawlik has joined #openstack-infra | 09:54 | |
| *** ianychoi_ has quit IRC | 09:56 | |
| *** abhishekk has quit IRC | 09:57 | |
| *** Xuchu has joined #openstack-infra | 09:58 | |
| *** jklare has quit IRC | 10:00 | |
| *** soniya29 has quit IRC | 10:00 | |
| *** gfidente has joined #openstack-infra | 10:03 | |
| *** jklare has joined #openstack-infra | 10:03 | |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** farhanjamil has quit IRC | 10:07 | |
| *** hashar has quit IRC | 10:15 | |
| *** hrw has joined #openstack-infra | 10:16 | |
| hrw | morning | 10:16 |
| hrw | Linaro's arm64 CI nodes are now in good shape and kolla ci jobs manage in ~1.5h when in past 3h were not enough. | 10:17 |
| hrw | but as there are just 8 arm64 nodes for whole ci I wonder is there a way to split in-project jobs to have separate 'normal zuul' + 'arm64 zuul' checks. | 10:18 |
| hrw | so developer sends patch to gerrit, both normal-zuul and arm64-zuul checks start but if arm64-zuul one sits in a queue then project still gets +1 from zuul | 10:18 |
| hrw | or do I overcomplicate thing? | 10:19 |
| *** dpawlik has quit IRC | 10:19 | |
| *** dpawlik has joined #openstack-infra | 10:22 | |
| *** ykarel is now known as ykarel|afk | 10:51 | |
| *** jklare has quit IRC | 10:57 | |
| *** jklare has joined #openstack-infra | 10:59 | |
| *** udesale has quit IRC | 11:04 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Update bindep for RHEL/CentOS 8 https://review.opendev.org/698448 | 11:15 |
| rishabhhpe | Hi , Can i know how to build a job template for zuul v3 for triple O configuration using devstack .. Please share if any link is there | 11:17 |
| *** dpawlik has quit IRC | 11:21 | |
| kevinz | \o/ | 11:27 |
| tosky | rishabhhpe: uhm, either you use tripleo or devstack | 11:28 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Update bindep for RHEL/CentOS 8 https://review.opendev.org/698448 | 11:30 |
| tosky | rishabhhpe: or maybe I misunderstood your question | 11:31 |
| *** arxcruz is now known as arxcruz|pto | 11:32 | |
| rishabhhpe | tosky: we are using devstack only on the same devstack we are spawning an instance on which we are installing devstack again and running the tempest ... so for that setup i need a job template | 11:32 |
| tosky | so no tripleo? | 11:34 |
| *** udesale has joined #openstack-infra | 11:35 | |
| *** pcaruana has joined #openstack-infra | 11:36 | |
| *** rishabhhpe has quit IRC | 11:37 | |
| *** ykarel|afk is now known as ykarel | 11:40 | |
| *** ianychoi has joined #openstack-infra | 11:49 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:52 | |
| *** ykarel is now known as ykarel|afk | 11:57 | |
| *** jklare has quit IRC | 11:57 | |
| *** dpawlik has joined #openstack-infra | 12:00 | |
| *** jklare has joined #openstack-infra | 12:00 | |
| *** larainema has quit IRC | 12:01 | |
| *** spsurya has joined #openstack-infra | 12:05 | |
| *** rfolco has joined #openstack-infra | 12:05 | |
| *** dpawlik has quit IRC | 12:05 | |
| *** nicolasbock has joined #openstack-infra | 12:18 | |
| *** dpawlik has joined #openstack-infra | 12:20 | |
| *** Xuchu has quit IRC | 12:25 | |
| *** dpawlik has quit IRC | 12:25 | |
| *** elod has quit IRC | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** dpawlik has joined #openstack-infra | 12:33 | |
| *** ykarel|afk is now known as ykarel| | 12:36 | |
| *** ykarel| is now known as ykarel | 12:36 | |
| *** ociuhandu has quit IRC | 12:37 | |
| *** ramishra has quit IRC | 12:39 | |
| *** udesale has quit IRC | 12:50 | |
| *** elod has joined #openstack-infra | 12:51 | |
| *** ykarel is now known as ykarel|afk | 12:56 | |
| *** jklare has quit IRC | 12:59 | |
| *** ykarel|afk has quit IRC | 13:00 | |
| *** rishabhhpe has joined #openstack-infra | 13:03 | |
| *** ociuhandu has joined #openstack-infra | 13:03 | |
| *** jklare has joined #openstack-infra | 13:04 | |
| rishabhhpe | tosky: yes no triple o | 13:04 |
| openstackgerrit | Stephen Finucane proposed openstack/hacking master: General cleanups https://review.opendev.org/651566 | 13:10 |
| openstackgerrit | Stephen Finucane proposed openstack/hacking master: Bump flake8 version to something modern https://review.opendev.org/651565 | 13:10 |
| openstackgerrit | Stephen Finucane proposed openstack/hacking master: WIP: Migrate tests https://review.opendev.org/698473 | 13:10 |
| *** goldyfruit_ has joined #openstack-infra | 13:17 | |
| *** SurajPatil has joined #openstack-infra | 13:18 | |
| *** rlandy has joined #openstack-infra | 13:19 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 13:20 |
| *** surpatil has quit IRC | 13:21 | |
| *** witek has quit IRC | 13:22 | |
| *** soniya29 has joined #openstack-infra | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:30 | |
| *** ykarel|afk has joined #openstack-infra | 13:31 | |
| *** surpatil has joined #openstack-infra | 13:32 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 13:33 |
| *** SurajPatil has quit IRC | 13:35 | |
| *** rh-jelabarre has joined #openstack-infra | 13:37 | |
| *** Lucas_Gray has joined #openstack-infra | 13:43 | |
| *** ykarel|afk is now known as ykarel | 13:46 | |
| *** Lucas_Gray has quit IRC | 13:50 | |
| *** Lucas_Gray has joined #openstack-infra | 13:51 | |
| *** soniya29 has quit IRC | 13:54 | |
| *** jklare has quit IRC | 13:58 | |
| *** mriedem has joined #openstack-infra | 13:59 | |
| *** bnemec has joined #openstack-infra | 14:01 | |
| *** jklare has joined #openstack-infra | 14:01 | |
| *** ociuhandu_ has joined #openstack-infra | 14:02 | |
| *** goldyfruit_ has quit IRC | 14:02 | |
| *** ociuhandu has quit IRC | 14:05 | |
| *** rishabhhpe has quit IRC | 14:05 | |
| *** kaisers has joined #openstack-infra | 14:06 | |
| *** Xuchu has joined #openstack-infra | 14:08 | |
| zbr | i decided to take my bsdm relationship with pip to a new level, https://discuss.python.org/t/can-we-finally-add-a-minimal-api-to-pip/2833 -- not sure if anything good will come out of that but i need to try. | 14:10 |
| *** ociuhandu_ has quit IRC | 14:20 | |
| *** rishabhhpe has joined #openstack-infra | 14:22 | |
| *** jamesmcarthur has joined #openstack-infra | 14:31 | |
| *** jamesmcarthur has quit IRC | 14:36 | |
| *** jamesmcarthur has joined #openstack-infra | 14:44 | |
| *** dtantsur is now known as dtantsur|brb | 14:44 | |
| *** ociuhandu has joined #openstack-infra | 14:46 | |
| *** eharney has quit IRC | 14:47 | |
| mordred | clarkb: the USDA is wrong. you can totally cook it to 130F/55C as long as you cook it for 2 hours after reaching that core temp. of course - the texture doing that is unlikely to be very pleasing to anyone | 14:52 |
| mordred | (actually, the USDA has very complete charts that discuss pasteurization times at different temps) | 14:56 |
| *** ociuhandu has quit IRC | 14:56 | |
| *** jklare has quit IRC | 14:56 | |
| persia | mordred: Do you have a URL for those? I was looking and on.ly found the "minimum safe" levels today, for 3 minute rest times (which are an improvement over the instant-ready temperatures that were previously posted in the summary tables) | 14:57 |
| *** ramishra has joined #openstack-infra | 15:00 | |
| *** jklare has joined #openstack-infra | 15:00 | |
| mordred | persia: there's a brief summary in https://www.seriouseats.com/2014/11/food-lab-sous-vide-turkey-crisp-skin-sous-vide-101-thanksgiving.html - but his links to the USDA tables themselves are now broken (and I haven't tried figuring out the new locations based on the old urls yet) | 15:00 |
| *** ramishra has quit IRC | 15:01 | |
| *** ramishra has joined #openstack-infra | 15:01 | |
| mordred | persia: https://www.fsis.usda.gov/wps/wcm/connect/bf3f01a1-a0b7-4902-a2df-a87c73d1b633/Salmonella-Compliance-Guideline-SVSP-RTE-Appendix-A.pdf?MOD=AJPERES | 15:01 |
| persia | Ooh, thanks. | 15:02 |
| *** jamesmcarthur has quit IRC | 15:06 | |
| * mordred has been useful today | 15:06 | |
| *** ociuhandu has joined #openstack-infra | 15:07 | |
| *** Goneri has joined #openstack-infra | 15:11 | |
| *** ociuhandu has quit IRC | 15:12 | |
| *** dpawlik has quit IRC | 15:16 | |
| *** eharney has joined #openstack-infra | 15:23 | |
| *** ociuhandu has joined #openstack-infra | 15:24 | |
| *** jamesmcarthur has joined #openstack-infra | 15:25 | |
| *** jamesmcarthur_ has joined #openstack-infra | 15:27 | |
| *** ociuhandu has quit IRC | 15:28 | |
| *** jamesmcarthur has quit IRC | 15:31 | |
| *** lpetrut has quit IRC | 15:32 | |
| *** surpatil has quit IRC | 15:33 | |
| *** panda is now known as panda|bbl | 15:44 | |
| *** chkumar|ruck is now known as raukadah | 15:47 | |
| *** Lucas_Gray has quit IRC | 15:54 | |
| *** jklare has quit IRC | 15:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Deprecate openstack-python-jobs templates https://review.opendev.org/698519 | 15:58 |
| clarkb | aiui previous recommendation was 180 for thighs which seems crazy | 15:59 |
| *** jklare has joined #openstack-infra | 16:00 | |
| mordred | clarkb: I cook dark meat to 74C/165F for texture reasons - dark meat at 65C/149F (my white meat temp) has a more unpleasant mouthfeel | 16:02 |
| mordred | 180 is an absurd temp | 16:02 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Deprecate openstack-python-jobs templates https://review.opendev.org/698519 | 16:07 |
| *** rishabhhpe has quit IRC | 16:09 | |
| *** lbragstad_ has joined #openstack-infra | 16:12 | |
| *** Lucas_Gray has joined #openstack-infra | 16:13 | |
| *** lbragstad has quit IRC | 16:15 | |
| *** witek has joined #openstack-infra | 16:17 | |
| *** Xuchu has quit IRC | 16:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Deprecate openstack-python-jobs templates https://review.opendev.org/698519 | 16:20 |
| *** rishabhhpe has joined #openstack-infra | 16:21 | |
| fungi | i'm gonna head out to lunch. probably have some very underdone tuna steak | 16:21 |
| * fungi thumbs his nose at the fda | 16:21 | |
| fungi | bbiaw | 16:21 |
| *** jaosorior has joined #openstack-infra | 16:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Deprecate openstack-python-jobs templates https://review.opendev.org/698519 | 16:28 |
| *** sean-k-mooney has joined #openstack-infra | 16:36 | |
| sean-k-mooney | o/ | 16:37 |
| sean-k-mooney | has anyone else had issue with the discuss list in the last few days | 16:37 |
| sean-k-mooney | http://paste.openstack.org/show/787461/ | 16:37 |
| sean-k-mooney | im getting a message saying that at least some messages are not being delivered | 16:38 |
| clarkb | sean-k-mooney: sensing via red hat mail system? | 16:43 |
| clarkb | I believe mordred and corvus escalated that to RH mail admins internally after zbr sshnaidm ran into that | 16:43 |
| clarkb | hrw: we can do that via a second pipeline | 16:44 |
| sshnaidm | sean-k-mooney, if it's from RH mail, then yes, we opened a ticket to IT | 16:45 |
| hrw | clarkb: any examples? | 16:45 |
| hrw | clarkb: and does it make sense? | 16:45 |
| mordred | the issue either way seems to be related to Mimecast who seem to, from what we can tell, ignore the fact that you don't need an MX record in this context | 16:45 |
| *** ricolin has quit IRC | 16:46 | |
| mordred | the internal RH IT people are theoretically discussing something with the mimecast people | 16:46 |
| sean-k-mooney | clarkb: we use gmail so googles | 16:46 |
| mordred | but I haven't heard a followup | 16:46 |
| openstackgerrit | Merged zuul/zuul master: Add additional info for executor.merge_jobs release note https://review.opendev.org/698324 | 16:46 |
| clarkb | hrw https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L254 the experimental pipeline is similar | 16:46 |
| hrw | thanks | 16:46 |
| mordred | sean-k-mooney: rh is also google/gmail - so the underlying issue is likely the same | 16:46 |
| sean-k-mooney | well its form my redhat email | 16:47 |
| mordred | oh - gotcha | 16:47 |
| mordred | so - it's almost CERTAINLY the same then :) | 16:47 |
| sean-k-mooney | but its sent via the gmail account not the other inteal legacy email sysmtem that it cant get peopel to give up | 16:47 |
| clarkb | sorry earlier message was supposed to be zbr or sshnaidm I guess it was sshnaidm. sorry for the noise zbr | 16:47 |
| zbr | nopb, i figured it out. funny i got some ML issues with podman recently, unrelated to this. | 16:48 |
| clarkb | should we add an MX record? | 16:48 |
| sean-k-mooney | mordred: im not really sure how this is a redhat issue however. surely its a google/mimecast issue no? | 16:48 |
| corvus | that was legal-discuss though, not openstack-discuss, so this represents a widening of the problem scope. | 16:48 |
| mordred | sean-k-mooney: yeah. rh it are our best point of contact | 16:49 |
| openstackgerrit | Stephen Finucane proposed openstack/hacking master: Bump flake8 version to something modern https://review.opendev.org/651565 | 16:49 |
| sean-k-mooney | mordred: ah gotch ya | 16:49 |
| mordred | corvus: yah. | 16:49 |
| hrw | clarkb: and then I mark jobs as pipeline 'check-arm64' in project's zuul files, right? | 16:50 |
| mordred | corvus, clarkb: given the widening, it makes me a bit more inclined to add an MX record even though that's not actually required - if this is a new way that google is breaking email from history it's probably unlikely it'll change :( | 16:50 |
| mordred | (what I really want to do is send a bunch of angry tweets - but I'm not sure that will accomplish anything) | 16:50 |
| stephenfin | who can I annoy to get https://review.opendev.org/#/c/651565/ merged and a 2.0 release of hacking cut? | 16:51 |
| stephenfin | once it comes back green from zuul, obviously | 16:51 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: New project: zuul-airship-roles https://review.opendev.org/698114 | 16:51 |
| mordred | stephenfin: most projects out there are pinned WRT versions, right? so making a release isn't going to immediately bork everybody | 16:52 |
| mordred | ? | 16:52 |
| Shrews | mordred: there is already enough rage on twitter making it less and less attractive | 16:52 |
| stephenfin | correct | 16:52 |
| clarkb | hrw: yes. Though we might be able to have a more generic pipeline for your use case. "slow-check" or whatever | 16:53 |
| sean-k-mooney | mordred: even if it was not fully pinned i would how it was at least pinned to the major version for anyone using it | 16:53 |
| clarkb | hrw: but ya that would be the process. Add new pipeline to the file I linked. Use pipeline in your projects | 16:53 |
| sean-k-mooney | so a 2.0 release really should be opt in | 16:53 |
| stephenfin | and it should be transparent for anyone not writing their own checks. Those guys (cough...nova...cough) need to fix stuff | 16:53 |
| clarkb | mordred: it would help if IT companies continued to run their own mail systems in order to ensure that standards still mean something | 16:54 |
| mordred | clarkb: it would - but they've all been sold on the lie "email is hard to run" | 16:54 |
| hrw | clarkb: thanks. will go back to it once we merge arm64 ci jobs and check how it goes | 16:54 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: New project: go-redfish https://review.opendev.org/698115 | 16:54 |
| clarkb | (I don't really have anything to stand on though as every IT org I've worked for has eventually given up and let someone else do it) | 16:54 |
| sean-k-mooney | clarkb: we still do because half the company refused to move | 16:54 |
| mordred | clarkb: not *every* - this one hasn't | 16:55 |
| sean-k-mooney | im sure there email is fine | 16:55 |
| hrw | clarkb: may not be needed as there is a chance for some new arm64 nodes during this/next month | 16:55 |
| clarkb | mordred: true. I guess I meant traditional IT org then :) | 16:55 |
| *** gyee has joined #openstack-infra | 16:55 | |
| mordred | clarkb: ++ | 16:55 |
| *** dtantsur|brb is now known as dtantsur | 16:55 | |
| clarkb | sean-k-mooney: sounds like that half may have been on to something :) | 16:55 |
| sean-k-mooney | clarkb: id stoped allowing new joiners to create an account on the old system about 6 months before i joined | 16:56 |
| clarkb | I'm not opposed to adding an MX record. It should be functionality equivalent to what we alredy have for standard compliant smtp speakers and may fix the googles | 16:56 |
| *** iurygregory has quit IRC | 16:56 | |
| sean-k-mooney | clarkb: what will you be pointing the mx record too | 16:56 |
| corvus | mordred: hrm, can we wait a bit to see what they come back with? "Domain has no MX records or is invalid" is still a bit vague. i agree it points in a direction of some voodoo we could do in an attempt to appease the mail gods, but personally i'd like to give a little more time for confirmation before we do that. | 16:56 |
| mordred | corvus: totes. I agree- I'd like to understand the problem | 16:57 |
| clarkb | sean-k-mooney: same location mail should already go to. But I'm happy to wait if that is corvus preference | 16:57 |
| mordred | I thnk "add an MX record" is our mitigation if we don't get anywhere - or the scope the carnage increases to the point where waiting starts to be untennable | 16:57 |
| corvus | i imagine we would point the mx record at itself. that should be fine. but i could totally imagine a world where we start getting ndrs that say "MX record points to self" :) | 16:58 |
| corvus | mordred: yep | 16:58 |
| corvus | mordred: maybe we could send a reply to the IT ticket saying "we have another incidence..." | 16:58 |
| mordred | corvus: ++ | 16:58 |
| mordred | I'll go do that | 16:58 |
| corvus | i'll do that? | 16:58 |
| corvus | mordred: you win | 16:58 |
| sean-k-mooney | well the mx record is ment to point to the mail server(s) that host that email domain | 16:59 |
| sean-k-mooney | so without one im a little confused how it gets delivered unless you have an upstream mx server for the domain its falling back to for the lists subdomain | 16:59 |
| clarkb | sean-k-mooney: if there is no MX record you are supposed to use the A/AAAA record | 17:00 |
| clarkb | which is how this is working for everyone but RH | 17:00 |
| *** jklare has quit IRC | 17:00 | |
| sean-k-mooney | for what its worth this only started to be an issue in the last week | 17:00 |
| sean-k-mooney | and ok that makes sense i guess | 17:00 |
| corvus | sean-k-mooney: https://tools.ietf.org/html/rfc5321#section-5.1 second paragraph starting with "If an empty list..." | 17:02 |
| mordred | sean-k-mooney: yeah - list.o.o has never had an MX record - so my guess is that mimecast made a change, or rh/google made a change to add/update mimecast | 17:02 |
| mordred | sean-k-mooney: is it happening for *all* of your emails to -discuss? or just occasionalluy? | 17:02 |
| mordred | (and would you like me to add you to the ticket?) | 17:03 |
| *** jklare has joined #openstack-infra | 17:03 | |
| corvus | in the case of sshnaidm, it happened for legal-discuss but not openstack-discuss | 17:03 |
| sshnaidm | I think IT got the problem, but maybe another case will make it more urgent | 17:04 |
| sean-k-mooney | ill check if any of my replies in the last day or so are in the archive | 17:04 |
| sean-k-mooney | im just getting the bounce email periodicly | 17:05 |
| corvus | knowing there's another affected address could be useful info for them | 17:05 |
| sean-k-mooney | hum so it look like nothing i have sent in the last week is there | 17:06 |
| mordred | *awesome* | 17:07 |
| mordred | so all of them are going to /dev/null, but you're only getting bounces for some of them | 17:07 |
| *** dougwig has quit IRC | 17:08 | |
| corvus | well, "red hat can't participate in one of their core upstream projects" is also maybe useful info for the it folks to help prioritize their work | 17:08 |
| *** dougwig has joined #openstack-infra | 17:08 | |
| sean-k-mooney | yep so that entire discussion we had last week or the week before about enabling the random number generate is missing | 17:08 |
| *** lucasagomes has quit IRC | 17:08 | |
| sean-k-mooney | well i think that was with clarkb? instead | 17:09 |
| sean-k-mooney | i dont drop people from the too line when i reply so the only mail message form me will be burried in the reply form others | 17:10 |
| mordred | https://etherpad.openstack.org/p/rJjAsDCMWr <-- corvus, sean-k-mooney how's the look for a ticket followup? | 17:11 |
| *** ykarel is now known as ykarel|away | 17:11 | |
| corvus | mordred: who's chris? last msg i see is from myles? | 17:11 |
| sean-k-mooney | ill check if i have got notificaitons for the others | 17:12 |
| sean-k-mooney | i assumed it was the same message so i might have | 17:12 |
| mordred | corvus: Chris Johnson is the last person who replied to the ticket | 17:12 |
| sean-k-mooney | ok i have only got two notifiaction | 17:12 |
| corvus | weird, did it not send an email for that?.... hrm | 17:13 |
| sean-k-mooney | but i have "sent" maybe 6 mails in that time | 17:13 |
| mordred | the ticket says it sent you an email | 17:13 |
| *** rpittau is now known as rpittau|afk | 17:13 | |
| mordred | update the etherpad a smidge | 17:15 |
| corvus | mordred: wow, that's great. i have signed into the ticketing system and do see the msg from chris. i did not receive that messge by email. :) <irony/> | 17:16 |
| sean-k-mooney | ya so i got 2 notification out of 5 emails that are missing | 17:16 |
| corvus | mordred: i made a tiny change | 17:17 |
| *** davidlenwell has quit IRC | 17:18 | |
| *** davidlenwell has joined #openstack-infra | 17:18 | |
| openstackgerrit | Stephen Finucane proposed openstack/hacking master: Bump flake8 version to something modern https://review.opendev.org/651565 | 17:18 |
| openstackgerrit | Stephen Finucane proposed openstack/hacking master: Address flake8 3.x violations https://review.opendev.org/698548 | 17:18 |
| corvus | (since, technically, i don't think any smtp rejections have happened, so i didn't want to introduce potential confusion) | 17:18 |
| stephenfin | mordred: This additional patch fixes the new violations flake8 3.x identifies. CI was green otherwise https://review.opendev.org/698548 | 17:19 |
| *** psachin has quit IRC | 17:19 | |
| *** panda|bbl is now known as panda | 17:20 | |
| sean-k-mooney | stephenfin: since you are here have you sent anyting to the openstack-discuss list in december | 17:21 |
| stephenfin | not that I can recall, no | 17:22 |
| stephenfin | though I was about to | 17:22 |
| sean-k-mooney | ok let use know when you do | 17:22 |
| mordred | sean-k-mooney: ticket update sent - and you're on the ticket now | 17:25 |
| corvus | i'm looking at my personal archives of openstack-discuss | 17:25 |
| *** rlandy is now known as rlandy|brb | 17:26 | |
| sean-k-mooney | mordred: cool thanks | 17:26 |
| corvus | the last message i see from a redhat.com address is from alex schultz on dec 5 | 17:27 |
| sean-k-mooney | corvus: i have some of the missing mails in my openstack-discuss folder but i dont know is that just a copy from my sent folder or not | 17:27 |
| corvus | take that with a grain of salt... i may not be doing that search quite right... | 17:27 |
| corvus | this message: http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011424.html | 17:27 |
| sean-k-mooney | the last one i have that is not me is also the 5th | 17:28 |
| clarkb | as another data point there are gmail emails from today | 17:29 |
| corvus | does anyone see a message after "[ptl] Train cycle-trailing release deadline Alex Schultz " from a red hatter on this list? http://lists.openstack.org/pipermail/openstack-discuss/2019-December/date.html | 17:29 |
| clarkb | regular gmail, not rh gmail | 17:29 |
| corvus | clarkb: yeah, i discovered https://www.mimecast.com/solutions/platforms/g-suite/ | 17:29 |
| *** ociuhandu has joined #openstack-infra | 17:30 | |
| clarkb | neat | 17:30 |
| corvus | clarkb: i suspect that rh has opted to add in mimecast to their gsuite pipeline, so all incoming and outgoing mail *also* goes through mimecast | 17:30 |
| corvus | here are the received headers from the last email we got: http://paste.openstack.org/show/787464/ note, it *does* include mimecast | 17:31 |
| clarkb | corvus: sshnaidm sent an email on December 5 after alex | 17:31 |
| clarkb | corvus: but that is the last one I see | 17:31 |
| clarkb | [all][tripleo][openstack-ansible] Openstack Ansible modules - next steps | 17:32 |
| corvus | ah yep, so my search was probably pulling up thread starts | 17:32 |
| mordred | so - the theory we have now is that rh added mimecast, and when they did rh employees ceased being able to send mails to the openstack mailing list | 17:33 |
| corvus | going back to late november (random sample at now 22), i also see mimecast headers | 17:34 |
| clarkb | mordred: or mimecast updated it which seems more likely given ^ | 17:34 |
| corvus | so i think mimecast may have been in the mix for a while, so this is more likely that ^ | 17:34 |
| mordred | nod | 17:34 |
| clarkb | *updated its rules | 17:34 |
| * mordred thinks pinging the openstack engineering head at rh is also likely a good idea - doing that now | 17:35 | |
| corvus | mordred: ++ | 17:36 |
| corvus | based on my archives, it looks like mimecast may have been added around 21 sep. and broken around 5 dec. | 17:39 |
| stephenfin | sean-k-mooney: done (subject: "Re: [dev] Upgrading flake8 to support f-strings") | 17:40 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Revert "Set default-ansible-version to 2.7 for all tenant (minus zuul)" https://review.opendev.org/698554 | 17:41 |
| sean-k-mooney | stephenfin: it might take a minute to propagate but ill keep an eye out for it | 17:42 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Remove zuul tenant ansible version pin https://review.opendev.org/698555 | 17:42 |
| clarkb | corvus: ^ | 17:42 |
| *** igordc has joined #openstack-infra | 17:42 | |
| *** hashar has joined #openstack-infra | 17:43 | |
| sean-k-mooney | stephenfin: i have got a different email from the list since you sent that but not that one | 17:44 |
| corvus | clarkb: thx, mordred: https://review.opendev.org/698554 and child? | 17:44 |
| *** diablo_rojo has joined #openstack-infra | 17:44 | |
| mordred | corvus: +A | 17:45 |
| *** ociuhandu has quit IRC | 17:45 | |
| mordred | stephenfin: oh wow. the core team on hacking is ... almost all people who don't work on openstack anym,ore | 17:46 |
| stephenfin | yuuuup | 17:46 |
| clarkb | is jogo still in it? | 17:46 |
| mordred | yup. | 17:46 |
| mordred | and julien danjou and sdague and dhellmann and dims | 17:46 |
| stephenfin | we should probably try move anything broadly useful to pycodestyle and kill the rest | 17:47 |
| mordred | bnemec is the only non-infra-core person who's still involved | 17:47 |
| sean-k-mooney | stephenfin: so its been about 5-7 minuts since you sent it and noting in my inbox or on the mail archive | 17:47 |
| clarkb | stephenfin: fwiw the infra team has disabled hacking checks on msot of its projects as they often seem too strict for us. That said I can't speak to the value of those checks for eg nova | 17:48 |
| mordred | I love them on sdk | 17:48 |
| mordred | but - that's an anecdote, not data - nor a basis for policy of any sort | 17:48 |
| stephenfin | sean-k-mooney: Weird. Is there something funky with RH Gmail or? | 17:48 |
| mordred | stephenfin: yes. it basically got broken a week ago, best we can tell - we've got a ticket open | 17:49 |
| sean-k-mooney | yes | 17:49 |
| stephenfin | delightful | 17:49 |
| *** rlandy|brb is now known as rlandy | 17:50 | |
| *** ykarel|away has quit IRC | 17:51 | |
| stephenfin | clarkb: Aye, very much project/individual specific, I imagine. It does help set a minimum bar for drive by contributions though, of which we get a few | 17:51 |
| mordred | stephenfin: well - at the very least if we're going to continue to maintain and use hacking, it should likely grow a core team who is interested in reviewing patches when they're needed - and not too eager to add new features (overly active hacking development has been a problem in the past and isn't super helpful) | 17:52 |
| stephenfin | we could, or we could decide that what's there is a good enough and won't need anything more than maintenance fixes, a lá git-review | 17:54 |
| stephenfin | *we won't need/accept | 17:54 |
| mordred | yah | 17:55 |
| mordred | like - the one you have up I think is good - and I think we shoudl land it | 17:55 |
| clarkb | iirc there is the concern that flake8 doesn't support the same plugin interface though? | 17:55 |
| mordred | it's the thign that is needed to move forward into the world of python3 more effectively | 17:55 |
| openstackgerrit | Merged openstack/hacking master: General cleanups https://review.opendev.org/651566 | 17:55 |
| clarkb | so if we wnt to keep getting upstream linter goodness hcaking might need updating? | 17:55 |
| clarkb | (or we can run the two tools separately) | 17:55 |
| stephenfin | clarkb: that's the thing I had mordred review above | 17:55 |
| clarkb | ah | 17:55 |
| *** gfidente has quit IRC | 17:56 | |
| mordred | clarkb: it has a new plugin interface - so I think part of a project updating to hacking v2 that project would have to either rewrite local plugins to the new interface, or would need to just drop local checks | 17:56 |
| stephenfin | yup | 17:56 |
| stephenfin | turns out that's pretty easy to do, at least https://review.opendev.org/#/c/695733/ | 17:56 |
| openstackgerrit | Merged openstack/project-config master: Revert "Set default-ansible-version to 2.7 for all tenant (minus zuul)" https://review.opendev.org/698554 | 17:56 |
| mordred | stephenfin: I agree - that's non-terrible | 17:57 |
| stephenfin | yup | 17:57 |
| stephenfin | there's probably even a way to keep things working as they were, but I haven't been smart enough to figure that out yet | 17:58 |
| stephenfin | ...and boy, have I tried | 17:58 |
| *** jklare has quit IRC | 17:58 | |
| *** ijw has joined #openstack-infra | 17:59 | |
| *** ijw_ has joined #openstack-infra | 18:00 | |
| *** derekh has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack/project-config master: Remove zuul tenant ansible version pin https://review.opendev.org/698555 | 18:02 |
| *** jklare has joined #openstack-infra | 18:02 | |
| *** ijw has quit IRC | 18:04 | |
| *** jpena is now known as jpena|off | 18:06 | |
| openstackgerrit | Merged openstack/hacking master: Address flake8 3.x violations https://review.opendev.org/698548 | 18:08 |
| *** igordc has quit IRC | 18:13 | |
| *** igordc has joined #openstack-infra | 18:14 | |
| *** efried has left #openstack-infra | 18:18 | |
| *** efried has joined #openstack-infra | 18:18 | |
| fungi | you lot got talky while i was lunching | 18:18 |
| *** rlandy is now known as rlandy|brb | 18:18 | |
| efried | fungi: was it you who was telling me about the CDN being flaky and making it hard for me to get at CI results? | 18:19 |
| clarkb | efried: that was fungi and my guess at the time since others of us were able to access those same urls | 18:19 |
| efried | Okay, I'd like to pursue it further if possible, because it happens very frequently. | 18:20 |
| efried | who's "in charge" of the thing I see as rackcdn.com? | 18:20 |
| clarkb | efried: rackspace | 18:20 |
| clarkb | (it is part of their object storage service that publicly accessible objects must be served through their cdn) | 18:20 |
| efried | is there like a person I could contact? | 18:21 |
| efried | or would my best bet be the ML? | 18:21 |
| *** dtantsur is now known as dtantsur|afk | 18:23 | |
| *** ociuhandu has joined #openstack-infra | 18:24 | |
| *** igordc has quit IRC | 18:25 | |
| *** eernst has joined #openstack-infra | 18:27 | |
| *** jaosorior has quit IRC | 18:27 | |
| clarkb | looking at dns records for the rackcdn domains they appear hosted by akamai | 18:28 |
| clarkb | and do not provide ipv6 | 18:28 |
| *** ociuhandu has quit IRC | 18:29 | |
| clarkb | possible that akamai has a problem report portal? | 18:29 |
| fungi | yeah, i get the impression it's an akamai interface these days | 18:30 |
| clarkb | the other place to check is your ISP as often ISPs will have direct connections to content distributors | 18:30 |
| *** jamesmcarthur_ has quit IRC | 18:30 | |
| fungi | also many isps set up transparent caching proxies known as "web accelerators" which have a wonderful tendency to break and start returning wrong or missing content | 18:32 |
| *** kjackal has quit IRC | 18:32 | |
| fungi | because if their clients request a particular page hundreds or thousands of times a second in aggregate, the only want to pay their transit providers for a few of those | 18:33 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: admin REST API: zuul-web integration https://review.opendev.org/643536 | 18:35 |
| *** pkopec has quit IRC | 18:38 | |
| *** rishabhhpe has quit IRC | 18:41 | |
| *** tosky has quit IRC | 18:44 | |
| *** kjackal has joined #openstack-infra | 18:46 | |
| *** addyess has joined #openstack-infra | 18:46 | |
| clarkb | jobs starting in the last 5 minutes or so are running ansible 2.8 | 18:51 |
| fungi | nice. and tox jobs are succeeding (or do we know yet)? | 18:51 |
| clarkb | don't know yet | 18:52 |
| *** ralonsoh has quit IRC | 18:55 | |
| efried | fungi, clarkb: I bcc'ed you on an email to the support address I found on rax's website -- and it bounced immediately :( | 18:57 |
| efried | Generating server: 544258-IExch13.ror-uc.rackspace.com | 18:58 |
| efried | cloudfiles@rackspace.com | 18:58 |
| efried | CO1NAM05FT026.mail.protection.outlook.com | 18:58 |
| efried | Remote Server returned '550 5.4.1 Recipient address rejected: Access denied [CO1NAM05FT026.eop-nam05.prod.protection.outlook.com]' | 18:58 |
| efried | this is pretty freakin frustrating | 18:58 |
| clarkb | efried: what ip address does dns give you for 975c7b7453777699c26a-26cfc451d4618cee8c0e628e323364bb.ssl.cf1.rackcdn.com | 18:59 |
| clarkb | I wonder if they are using anycast of if it is dns load balanced | 18:59 |
| clarkb | we might be able to narrow the problem down to a specific IP if dns load balanced then go bug akamai | 19:00 |
| clarkb | if it is anycast we won't have enough info | 19:00 |
| efried | 975c7b7453777699c26a-26cfc451d4618cee8c0e628e323364bb.ssl.cf1.rackcdn.com is an alias for cf1.rackcdn.com.edgekey.net. | 19:00 |
| efried | cf1.rackcdn.com.edgekey.net is an alias for e4845.g.akamaiedge.net. | 19:00 |
| efried | e4845.g.akamaiedge.net has address 23.214.108.170 | 19:00 |
| efried | FWIW the URL I put in the email eventually started working for me (as described in the email) | 19:00 |
| clarkb | I get 96.17.4.171 | 19:00 |
| clarkb | so not anycast | 19:00 |
| clarkb | I'm seattle you are dallas | 19:02 |
| clarkb | they are geographically distinct too | 19:02 |
| clarkb | might be able to try support@akamai.com with that info? | 19:03 |
| clarkb | we aren't customers though so may get told to go away :/ | 19:03 |
| clarkb | fungi: the failures I'm seeing all appear to be legit so far | 19:04 |
| clarkb | and we have successes now so this is looking good | 19:04 |
| efried | I'm in Taylor, which is closer to Austin than Dallas. Do you mean that IP is from Dallas? | 19:05 |
| clarkb | efried: yes that IP resides in Dallas (roughly from a traceroute) | 19:05 |
| efried | ack | 19:05 |
| efried | I'll try resending to that email, can't hurt <shrug> | 19:06 |
| efried | thanks as always clarkb | 19:06 |
| clarkb | efried: we might need to edit ti a bit to be more akamai specific | 19:06 |
| clarkb | basically they won't care about openstack things but if we provide urls then say it doesn't work reliably at 23.214.108.170 but does at 96.17.4.171 that should give them something to look at | 19:07 |
| fungi | right, as cdn operators what they *will* generally care about is that content from one of their endpoints is consistent with content from the rest of their endpoints | 19:08 |
| efried | clarkb: that hostname you had me look up, was that from a random CI job? | 19:08 |
| clarkb | efried: yes | 19:08 |
| efried | okay. | 19:09 |
| clarkb | efried: hwoever its the same IP providing your example | 19:09 |
| efried | makes sense | 19:09 |
| clarkb | one option available to you is to hijack your local dns to provide different answers for those names | 19:10 |
| clarkb | however if akamai changes IPs you'll break until you sort that out so not a great choice | 19:11 |
| *** nicolasbock has quit IRC | 19:11 | |
| *** rlandy|brb is now known as rlandy | 19:11 | |
| *** nicolasbock has joined #openstack-infra | 19:11 | |
| efried | mm | 19:11 |
| efried | clarkb: where are you actually sitting? | 19:12 |
| clarkb | efried: edit /etc/hosts to have 96.17.4.171 e4845.g.akamaiedge.net. and 96.17.4.172 e4847.g.akamaiedge.net. | 19:12 |
| clarkb | efried: portland oregon | 19:12 |
| efried | okay | 19:12 |
| fungi | i'm sitting in an armchair in my living room | 19:12 |
| clarkb | efried: doing the etc/hosts hack may be worthwhile in the short term just to confirm you get reliable responses out of different akamai nodes | 19:13 |
| clarkb | efried: but long term risk is akamai changes IPs and things stop working | 19:13 |
| efried | clarkb: hopefully I'll notice and remember the hack :) | 19:14 |
| *** spsurya has quit IRC | 19:14 | |
| *** jtomasek has quit IRC | 19:17 | |
| openstackgerrit | Merged zuul/zuul-jobs master: openshift speculative containers https://review.opendev.org/696939 | 19:17 |
| *** kjackal has quit IRC | 19:18 | |
| *** jamesmcarthur has joined #openstack-infra | 19:20 | |
| efried | clarkb, fungi: second attempt sent | 19:21 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: New project: zuul-airship-roles https://review.opendev.org/698114 | 19:25 |
| *** goldyfruit_ has joined #openstack-infra | 19:26 | |
| *** witek has quit IRC | 19:30 | |
| *** eharney has quit IRC | 19:30 | |
| clarkb | efried: that looks good. Now we'll discover if they care about random reports from the internet not directly from paying customers | 19:30 |
| *** kjackal has joined #openstack-infra | 19:30 | |
| efried | clarkb: somebody's paying them? rax? Or no? | 19:31 |
| mordred | somebody somewhere has to be paying them right? | 19:31 |
| *** lpetrut has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has quit IRC | 19:34 | |
| *** iurygregory has joined #openstack-infra | 19:34 | |
| clarkb | efried: mordred rax I expect in this case | 19:34 |
| *** jamesmcarthur has joined #openstack-infra | 19:34 | |
| *** Lucas_Gray has quit IRC | 19:35 | |
| clarkb | akamai once created a dns loop and demanded I update our dns to fix it when I was a customer | 19:35 |
| clarkb | that was fun | 19:36 |
| clarkb | also the extent of my interaction wiht their support | 19:36 |
| *** mriedem has quit IRC | 19:37 | |
| ianw | hrw: just going through scrollback "so developer sends patch to gerrit, both normal-zuul and arm64-zuul checks start but if arm64-zuul one sits in a queue then project still gets +1 from zuul" | 19:40 |
| hrw | ianw: hi | 19:40 |
| ianw | that sort of sounds like an automatically triggered "experimental" queue | 19:41 |
| hrw | ianw: yes | 19:41 |
| clarkb | ianw: ya I linked the experimental pipelien config as being similar earlier | 19:41 |
| hrw | ianw: with 8 arm64 nodes I suspect long queues will be created | 19:41 |
| ianw | ahh, right, yes sorry reading further i see that now :) | 19:42 |
| hrw | ;) | 19:42 |
| pabelanger | ianw: hrw: clarkb: in rdoproject, for 3pci (at one point) jobs would only run on new patchset, and zuul left +1 on patch. This was to avoid overloading downstream CI on patches that failed on zuul.o.o to start with. That needed a 2nd pipeline outside of check | 19:42 |
| ianw | yeah, i guess the only question is, much like "does a tree falling in the forest with nobody to hear it make a sound" ... "does a queue not voting actually catch any problems" :) | 19:43 |
| hrw | ianw: depends on how loud the lumberjack is | 19:44 |
| ianw | i also have arm64 tests for diskimage-builder and our own system-config (for setting up mirrors) that i haven't merged, mostly due to similar concerns over holding up the gate with small resources | 19:45 |
| clarkb | hrw: ianw I lived near https://en.wikipedia.org/wiki/Statue_of_Paul_Bunyan_(Portland,_Oregon) a few years back | 19:46 |
| *** tesseract has quit IRC | 19:46 | |
| clarkb | a very large but not very loud lumberjack | 19:46 |
| *** lpetrut has quit IRC | 19:46 | |
| hrw | ianw: I think that separate arm64 pipeline which would need to be called by hand could be a good start. until we get more nodes | 19:48 |
| clarkb | hrw: you should be able to use the experimental pipeline for that today if you don't mine triggering it by hand | 19:49 |
| hrw | clarkb: will go that way probably | 19:49 |
| ianw | hrw: yeah, although i think we probably have a case with several different projects wanting similar speculative behaviour for a separate queue | 19:49 |
| ianw | clarkb: best i can do is https://en.wikipedia.org/wiki/Big_Banana, but that's like 8 hours drive from me :) | 19:50 |
| hrw | ianw: does not surprise me | 19:50 |
| clarkb | ianw: I love that https://en.wikipedia.org/wiki/Australia%27s_big_things is a thing | 19:50 |
| clarkb | and one of them is a big can of tooheys new | 19:51 |
| ianw | hrw: the place to define it would likely be https://opendev.org/opendev/project-config/src/branch/master/zuul.d/pipelines.yaml | 19:57 |
| ianw | clarkb: yes, well stereotypes always have a bit of truth :) | 19:57 |
| ianw | it wouldn't actually enforce that arm64 jobs were the only thing in there, but it would be the convention. we can see what others might think | 19:58 |
| otherwiseguy_ | question re: tempest/neutron-tempest-plugin and https://bugs.launchpad.net/tripleo/+bug/1855985. I'm currently seeing some failures due to the test_agent_management code stepping on itself because both the neutron-tempest-plugin version and the tempest in-tree version are running and modifying the agent. Anyone know of any reason not to remove the in-tempest-tree versions? | 19:59 |
| openstack | Launchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed] | 19:59 |
| *** jklare has quit IRC | 19:59 | |
| *** jklare has joined #openstack-infra | 20:02 | |
| clarkb | otherwiseguy_: the qa team might be better positioned to answer that given it is tempest stuff. I don't personally know the answer | 20:03 |
| *** lbragstad has joined #openstack-infra | 20:03 | |
| otherwiseguy_ | clarkb: thanks! | 20:03 |
| *** harlowja has joined #openstack-infra | 20:07 | |
| *** ociuhandu has joined #openstack-infra | 20:09 | |
| *** eharney has joined #openstack-infra | 20:12 | |
| * hrw off | 20:17 | |
| *** ociuhandu has quit IRC | 20:21 | |
| *** ociuhandu has joined #openstack-infra | 20:21 | |
| clarkb | fungi: that is a fun neutron + dhcp interaction you helped diagnose (I'm catching up on email now) | 20:31 |
| fungi | i haven't caught up on it, but if neutron is cheerfully adding every network into the routes list for dhcp responses, that can only go on for so long before your dhcp response is unable to contain the sprawl | 20:37 |
| clarkb | yup haleyb confirmed that is the case and neutron may need to have checks in place | 20:39 |
| *** ociuhandu has quit IRC | 20:40 | |
| *** ociuhandu has joined #openstack-infra | 20:41 | |
| haleyb | yes, and there are possibly many ways to solve it, don't know which is better for the user | 20:41 |
| clarkb | a limit with documentation that routed networks will be necessary due to dhcp limitations seems fine | 20:41 |
| fungi | in the past i've solved similar issues by aggregating routes into larger netblocks/cidrs | 20:43 |
| fungi | just have to be careful you're not creating black holes in the process | 20:44 |
| fungi | there is probably some way neutron apply some subnet math to work out which routes can be safely combined | 20:44 |
| fungi | er, some way neutron could apply some subnet math i mean | 20:45 |
| *** iurygregory has quit IRC | 20:45 | |
| fungi | if they're rfc 1918 networks and there's only one gateway to them from the guest's perspective, then that's fairly easy because those should never be accessible through a global internet gateway anyway | 20:46 |
| fungi | just slap 10/8 or 172.16/12 or 192.168/16 in the route list and call it a day | 20:47 |
| haleyb | yeah, if we could compress that would be best, but if it's over the limit there's no way to signal to the user, unless we want to add some field to the network | 20:50 |
| fungi | also unless neutron is directly constructing the dhcp options field, it can't know what other options are present which reduce available length for option 121 | 20:50 |
| fungi | and it's almost certainly not going to be party to dhcp response size negotiations with the clients to find out that it can safely increase it either | 20:51 |
| haleyb | right, we can only assume based on some rough values | 20:52 |
| *** pkopec has joined #openstack-infra | 20:52 | |
| fungi | i suppose you could provide a config option where the operator is able to directly specify a max length for option 121 so that they get a sane error back from neutron when trying to go beyond that rather than having it bubble up in the dhcpd | 20:53 |
| fungi | but that requires the operator to understand protocol nuances in the first place, which they typically won't, so sort of user-hostile | 20:54 |
| *** ociuhandu has quit IRC | 20:57 | |
| *** jklare has quit IRC | 20:57 | |
| haleyb | returning a 2xx status to a POST indicating a possible issue just isn't possible :( it would be better to fail if we're close | 20:58 |
| fungi | yeah | 20:59 |
| fungi | it's a tough position, there just isn't much of an api between neutron and the dhcpd (i guess it could be taken as a reason to implement a dhcpd directly in neutron, but you really don't want to deal with that special circle of hell) | 21:00 |
| *** jklare has joined #openstack-infra | 21:01 | |
| *** slaweq has quit IRC | 21:03 | |
| haleyb | OVN is a little closer to that possibly, i just don't know the details of that part of the code | 21:03 |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** ociuhandu has joined #openstack-infra | 21:13 | |
| *** slaweq has quit IRC | 21:16 | |
| *** hashar has quit IRC | 21:19 | |
| *** hashar has joined #openstack-infra | 21:22 | |
| haleyb | te | 21:22 |
| haleyb | oops | 21:22 |
| *** rcernin has joined #openstack-infra | 21:23 | |
| *** dkehn has joined #openstack-infra | 21:32 | |
| *** lbragstad has quit IRC | 21:34 | |
| *** mattw4 has joined #openstack-infra | 21:36 | |
| *** pcaruana has quit IRC | 21:38 | |
| efried | clarkb, fungi: strike two. akamai wants to authenticate me six ways from sunday before they'll even consider responding to my inquiry. | 21:39 |
| fungi | yeesh | 21:45 |
| fungi | clarkb's terrible /etc/hosts workaround seems less and less terrible by the second | 21:46 |
| *** ociuhandu has quit IRC | 21:48 | |
| *** ociuhandu has joined #openstack-infra | 21:52 | |
| *** jamesmcarthur has quit IRC | 21:56 | |
| *** jamesmcarthur has joined #openstack-infra | 21:58 | |
| *** jklare has quit IRC | 21:58 | |
| *** jklare has joined #openstack-infra | 22:01 | |
| *** ociuhandu has quit IRC | 22:06 | |
| *** ociuhandu has joined #openstack-infra | 22:06 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** tosky has joined #openstack-infra | 22:15 | |
| *** slaweq has quit IRC | 22:16 | |
| *** pkopec has quit IRC | 22:17 | |
| *** rfolco has quit IRC | 22:25 | |
| *** ociuhandu has quit IRC | 22:27 | |
| *** nicolasbock has quit IRC | 22:27 | |
| *** sshnaidm is now known as sshnaidm|afk | 22:33 | |
| *** mriedem has joined #openstack-infra | 22:37 | |
| *** rfolco has joined #openstack-infra | 22:45 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Set correct python version for non-chroot scripts https://review.opendev.org/697211 | 22:52 |
| *** hashar has quit IRC | 22:52 | |
| *** jamesmcarthur has quit IRC | 22:56 | |
| *** jklare has quit IRC | 22:59 | |
| *** jamesmcarthur has joined #openstack-infra | 23:03 | |
| *** jklare has joined #openstack-infra | 23:03 | |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** hwoarang has quit IRC | 23:07 | |
| *** hwoarang has joined #openstack-infra | 23:07 | |
| *** tkajinam has joined #openstack-infra | 23:08 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: zuul: arm64 pipeline https://review.opendev.org/698606 | 23:14 |
| ianw | hrw/clarkb: ^ something like that, maybe ... | 23:14 |
| openstackgerrit | Merged openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 23:15 |
| *** slaweq has quit IRC | 23:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Work around Trusty ext4 metadata_csum errors on Bionic https://review.opendev.org/698346 | 23:21 |
| clarkb | ianw: one note on the pipeline change | 23:21 |
| clarkb | I think there is a small optimization we should make | 23:21 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: zuul: arm64 pipeline https://review.opendev.org/698606 | 23:23 |
| ianw | good idea | 23:23 |
| *** rlandy is now known as rlandy|bbl | 23:27 | |
| *** hwoarang has quit IRC | 23:30 | |
| *** hwoarang has joined #openstack-infra | 23:31 | |
| *** slaweq has joined #openstack-infra | 23:32 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Replace skip with errors=ignore https://review.opendev.org/660541 | 23:35 |
| ianw | infra-root: ^ i think that with the merge of 2.9.1 to bridge.o.o, that will silence a bunch of those warnings | 23:36 |
| *** tosky has quit IRC | 23:40 | |
| *** slaweq has quit IRC | 23:41 | |
| *** eernst has quit IRC | 23:42 | |
| *** eernst has joined #openstack-infra | 23:44 | |
| *** eernst has quit IRC | 23:49 | |
| clarkb | I've confirmed that ianw's fix appears to be in 2.9.1 and 2.9.1 is installed on bridge | 23:50 |
| clarkb | TIL about the null mx record | 23:54 |
| clarkb | also we don't have a null mx record for lists.o.o which rules that potential problem out | 23:55 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: Add Google Cloud provider https://review.opendev.org/698342 | 23:57 |
| *** ccamacho|pto has quit IRC | 23:59 | |
| clarkb | https://community.mimecast.com/s/article/Monitoring-the-Delivery-Queue-2145229965 has a little more info on what mimecast's message means | 23:59 |
| clarkb | seems that they got an empty MX and that is an issue (doesn't really indicate if they also checked for A/AAAA) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!