| ianw | another thread, that looks stuck during reconfiguration -> http://paste.openstack.org/show/755089/ | 00:02 |
|---|---|---|
| clarkb | that isn't ringing any bells for me, but I'm known to forget things | 00:02 |
| ianw | yep my comment in https://review.opendev.org/#/c/665014/1 | 00:03 |
| *** betherly has joined #openstack-infra | 00:04 | |
| ianw | # ls /proc/3426/task/ | wc -l | 00:04 |
| ianw | 262 | 00:04 |
| ianw | at that time we suspected the failure of the control plane uploads | 00:05 |
| *** dchen has joined #openstack-infra | 00:05 | |
| ianw | it seems to have leveled out there, but i wonder if that's because it's basically deadlocked itself | 00:06 |
| clarkb | or cpu contention makes it slow down | 00:07 |
| clarkb | do we want to watch 01 and see if it is happy now? | 00:07 |
| clarkb | and if so we can also restart 02? | 00:07 |
| donnyd | I see some images coming in now | 00:07 |
| ianw | clarkb: yeah, but a lot of the threads seem to be waiting now on the GIL. so i wonder if it has stopped itself from spawning new threads through contention | 00:07 |
| clarkb | ianw: that would not surprise me | 00:08 |
| ianw | i feel like it must be that the zypper command has hung, but why that leads to nodepool death is an open question :/ | 00:08 |
| *** betherly has quit IRC | 00:09 | |
| openstackgerrit | Merged zuul/zuul master: Add log browsing to build page https://review.opendev.org/671906 | 00:10 |
| *** threestrands has joined #openstack-infra | 00:10 | |
| ianw | so i have gone in and run "cat /proc/23186/fd/1" and that zypper process resumed and ran | 00:12 |
| clarkb | it was waiting for something to read stdout? | 00:12 |
| ianw | yeah, i would say. i'm doing the same on the other one. let's see if nodepool comes alive when that one clears too... | 00:13 |
| fungi | gitea07 and 08 git cg finished | 00:15 |
| donnyd | my fans came in tonight for the nvme drives, so I am good to hook. However boot from volume needs to be set in nodepool. | 00:15 |
| clarkb | donnyd: oh are we making that change too? | 00:16 |
| donnyd | I would like to give it a whirl. I am hoping cinder will remain stable enough | 00:17 |
| donnyd | I am not using ceph though, so no need to change to raw images | 00:17 |
| clarkb | donnyd: we need to apply https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L266-L267 to all of the image entries at https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl02.openstack.org.yaml#L261-L325 | 00:17 |
| clarkb | donnyd: would you like to propose that change? | 00:17 |
| donnyd | yea I can take care of it | 00:18 |
| ianw | clarkb: ok, we also have from jul 25 a stuck process | 00:18 |
| ianw | File "/opt/dib_tmp/dib_build.GicrW4FK/hooks/extra-data.d/50-create-repo-list", line 41, in get_project_list | 00:18 |
| ianw | for f in yaml.load(urlopen(url)): | 00:18 |
| ianw | it goes on to be stuck in libssl somewhere ... now that could have been gitea instability given the date | 00:18 |
| clarkb | that would've been ya that | 00:18 |
| donnyd | so does it only need to be changed in nl02? | 00:19 |
| fungi | sounds likely | 00:19 |
| fungi | donnyd: yes, our nodepool launchers are pegged to specific providers | 00:19 |
| donnyd | thanks | 00:20 |
| *** pkopec has quit IRC | 00:20 | |
| fungi | (our nodepool image builders are not technically, but are architecture-specific) | 00:20 |
| clarkb | donnyd: and you'll probably update the flavor too? | 00:21 |
| clarkb | to a flavor without disk assigned to it? | 00:21 |
| *** diablo_rojo has joined #openstack-infra | 00:21 | |
| donnyd | ok, I can do that as well | 00:21 |
| donnyd | is that so nova won't worry about local disk space when launching? | 00:21 |
| openstackgerrit | Merged zuul/zuul master: Move artifacts to their own section https://review.opendev.org/672379 | 00:21 |
| clarkb | yes | 00:22 |
| ianw | after manually clearing blocked stdout on a few zypper processes, and killing some of those hung cache-url process, it's actually gone on building more images now | 00:22 |
| clarkb | otherwise I think it may still allocate a local disk | 00:22 |
| ianw | but there's still hundreads of threads. i'll just do a reboot at this point, i don't know if there's much more to glean | 00:23 |
| donnyd | clarkb: I am pretty sure you are correct | 00:23 |
| clarkb | ianw: wfm | 00:23 |
| *** betherly has joined #openstack-infra | 00:25 | |
| *** gyee has quit IRC | 00:28 | |
| *** betherly has quit IRC | 00:30 | |
| ianw | hrm, nb02 still appears down after reboot, which is not ideal | 00:31 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: set FN to boot from volume for Cinder NVME backend https://review.opendev.org/673378 | 00:32 |
| *** yamamoto has joined #openstack-infra | 00:32 | |
| openstackgerrit | Merged zuul/zuul master: Remember tab location on build page https://review.opendev.org/672836 | 00:34 |
| ianw | #status log nb02 rebooted after stuck processes sent nodepool-builder into deadlock | 00:41 |
| openstackstatus | ianw: finished logging | 00:41 |
| ianw | i had to externally reboot it, console was filled with dracut messages about "/dev/log" | 00:42 |
| *** betherly has joined #openstack-infra | 00:46 | |
| donnyd | will nodepool take care of the security group rules? | 00:50 |
| ianw | donnyd: i think that's something the cloud launcher roles would work out | 00:51 |
| *** betherly has quit IRC | 00:51 | |
| donnyd | ok | 00:53 |
| openstackgerrit | Merged zuul/zuul master: Use base 1 line number anchors in log view https://review.opendev.org/672837 | 00:54 |
| ianw | prometheanfire: do we need a dib release to get gentoo building with that new directory stuff? | 00:55 |
| donnyd | it also looks like only one image was loaded, but I am also going to assume since it was just rebooted it will be a while | 00:55 |
| *** ricolin_ has joined #openstack-infra | 00:58 | |
| *** gregoryo has joined #openstack-infra | 00:59 | |
| *** panda has quit IRC | 01:06 | |
| fungi | donnyd: our cloud launcher roles basically set an empty allow-all security group, and then iptables is configured on the node images to block inbound connections other than a handful of common service ports (and allow all egress connections) | 01:06 |
| fungi | clarkb has proposed switching that around last week | 01:06 |
| donnyd | I just couldn't remember if I set them or cloud-launcher did | 01:07 |
| donnyd | i just checked the v6 networking for external access in the openstackzuul project, looks to be good to go | 01:07 |
| *** panda has joined #openstack-infra | 01:08 | |
| fungi | yeah, they didn't get set until saturday when i was fiddling with cloud launcher to get the keys uploaded | 01:08 |
| fungi | since it's all the same batch of ansible tasks | 01:08 |
| openstackgerrit | Merged opendev/system-config master: Put gitea07 and gitea08 back into service https://review.opendev.org/673373 | 01:11 |
| openstackgerrit | Merged zuul/zuul master: Add severity filtering to logs https://review.opendev.org/672839 | 01:14 |
| *** igordc has quit IRC | 01:15 | |
| *** betherly has joined #openstack-infra | 01:18 | |
| *** betherly has quit IRC | 01:22 | |
| openstackgerrit | Merged zuul/zuul master: Colorize log severity https://review.opendev.org/673103 | 01:29 |
| openstackgerrit | Merged zuul/zuul master: Add raw links to log manifest https://review.opendev.org/673104 | 01:46 |
| prometheanfire | ianw: I think builds should still be working systemd is still on the old profile | 01:47 |
| ianw | ok, keep an eye, the builds were stuck on nb02 but per above likely as not gitea issues from days ago | 01:51 |
| prometheanfire | kk | 01:51 |
| openstackgerrit | Merged zuul/zuul master: Rename view to logfile https://review.opendev.org/673105 | 02:03 |
| *** yikun has joined #openstack-infra | 02:04 | |
| *** diablo_rojo has quit IRC | 02:06 | |
| *** ekultails has quit IRC | 02:15 | |
| *** tdasilva has quit IRC | 02:33 | |
| *** tdasilva has joined #openstack-infra | 02:33 | |
| openstackgerrit | Merged openstack/diskimage-builder master: update version of open-iscsi that is installed on musl https://review.opendev.org/672152 | 02:37 |
| *** hongbin has joined #openstack-infra | 02:53 | |
| *** auristor has quit IRC | 02:58 | |
| *** auristor has joined #openstack-infra | 03:01 | |
| *** tdasilva has quit IRC | 03:03 | |
| *** tdasilva has joined #openstack-infra | 03:03 | |
| *** auristor has quit IRC | 03:05 | |
| *** auristor has joined #openstack-infra | 03:08 | |
| *** auristor has quit IRC | 03:20 | |
| *** auristor has joined #openstack-infra | 03:25 | |
| *** threestrands has quit IRC | 03:30 | |
| *** threestrands has joined #openstack-infra | 03:30 | |
| *** auristor has quit IRC | 03:31 | |
| *** threestrands has quit IRC | 03:31 | |
| *** hongbin has quit IRC | 03:34 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** auristor has joined #openstack-infra | 03:45 | |
| *** ykarel has joined #openstack-infra | 03:57 | |
| *** ramishra has joined #openstack-infra | 04:00 | |
| *** tdasilva has quit IRC | 04:04 | |
| *** jaosorior has quit IRC | 04:05 | |
| *** tdasilva has joined #openstack-infra | 04:05 | |
| *** apetrich has quit IRC | 04:20 | |
| *** udesale has joined #openstack-infra | 04:23 | |
| *** gary_perkins has quit IRC | 04:32 | |
| *** gary_perkins has joined #openstack-infra | 04:33 | |
| *** whoami-rajat has joined #openstack-infra | 04:40 | |
| *** ykarel is now known as ykarel|afk | 04:40 | |
| *** ykarel|afk has quit IRC | 04:45 | |
| *** jaosorior has joined #openstack-infra | 05:04 | |
| *** tdasilva has quit IRC | 05:06 | |
| *** tdasilva has joined #openstack-infra | 05:06 | |
| *** ykarel|afk has joined #openstack-infra | 05:13 | |
| *** ykarel|afk is now known as ykarel | 05:13 | |
| *** ricolin__ has joined #openstack-infra | 05:21 | |
| *** ricolin_ has quit IRC | 05:24 | |
| *** cloudnull has quit IRC | 05:25 | |
| *** d34dh0r53 has quit IRC | 05:25 | |
| *** bgmccollum has quit IRC | 05:25 | |
| *** yamamoto has quit IRC | 05:26 | |
| *** yamamoto has joined #openstack-infra | 05:27 | |
| *** jamesmcarthur_ has quit IRC | 05:28 | |
| *** rascasoft has quit IRC | 05:30 | |
| *** kopecmartin|off is now known as kopecmartin | 05:48 | |
| *** shachar has quit IRC | 06:05 | |
| *** shachar has joined #openstack-infra | 06:05 | |
| *** tdasilva has quit IRC | 06:07 | |
| *** bgmccollum has joined #openstack-infra | 06:07 | |
| *** tdasilva has joined #openstack-infra | 06:07 | |
| *** cloudnull has joined #openstack-infra | 06:17 | |
| *** d34dh0r53 has joined #openstack-infra | 06:17 | |
| *** dpawlik has joined #openstack-infra | 06:33 | |
| *** dpawlik has quit IRC | 06:38 | |
| *** dpawlik has joined #openstack-infra | 06:43 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations https://review.opendev.org/652114 | 06:49 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --check-config option to zuul scheduler https://review.opendev.org/542160 | 06:49 |
| *** ginopc has joined #openstack-infra | 06:51 | |
| *** slaweq has joined #openstack-infra | 06:53 | |
| *** rpittau|afk is now known as rpittau | 06:58 | |
| *** yoctozepto has quit IRC | 07:00 | |
| *** jaosorior has quit IRC | 07:01 | |
| *** rcernin has quit IRC | 07:02 | |
| *** tdasilva has quit IRC | 07:08 | |
| *** pgaxatte has joined #openstack-infra | 07:08 | |
| *** tdasilva has joined #openstack-infra | 07:09 | |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** Goneri has joined #openstack-infra | 07:20 | |
| *** xek has joined #openstack-infra | 07:20 | |
| openstackgerrit | Merged openstack/diskimage-builder master: fix comments / spelling errors in gentoo element https://review.opendev.org/673295 | 07:22 |
| *** tosky has joined #openstack-infra | 07:24 | |
| *** jpena|off is now known as jpena | 07:24 | |
| *** pkopec has joined #openstack-infra | 07:28 | |
| *** aedc has joined #openstack-infra | 07:31 | |
| *** ykarel is now known as ykarel|lunch | 07:42 | |
| *** ricolin__ is now known as ricolin | 07:44 | |
| *** Goneri has quit IRC | 07:56 | |
| *** roman_g has joined #openstack-infra | 07:57 | |
| *** ralonsoh has joined #openstack-infra | 07:58 | |
| *** dtantsur|afk is now known as dtantsur | 08:07 | |
| *** priteau has joined #openstack-infra | 08:08 | |
| *** lucasagomes has joined #openstack-infra | 08:09 | |
| *** yoctozepto has joined #openstack-infra | 08:09 | |
| *** tdasilva has quit IRC | 08:10 | |
| *** tdasilva has joined #openstack-infra | 08:10 | |
| *** happyhemant has joined #openstack-infra | 08:12 | |
| *** aedc has quit IRC | 08:13 | |
| *** dchen has quit IRC | 08:16 | |
| *** lpetrut has joined #openstack-infra | 08:18 | |
| *** derekh has joined #openstack-infra | 08:27 | |
| *** gregoryo has quit IRC | 08:29 | |
| *** roman_g has quit IRC | 08:43 | |
| *** e0ne has joined #openstack-infra | 08:45 | |
| *** jaosorior has joined #openstack-infra | 08:45 | |
| *** ykarel|lunch is now known as ykarel| | 08:46 | |
| *** ykarel| is now known as ykarel | 08:46 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 08:46 |
| *** tdasilva_ has joined #openstack-infra | 08:46 | |
| *** tdasilva has quit IRC | 08:50 | |
| *** apetrich has joined #openstack-infra | 08:59 | |
| *** Goneri has joined #openstack-infra | 09:02 | |
| *** tkajinam has quit IRC | 09:03 | |
| *** SotK has quit IRC | 09:06 | |
| *** SotK has joined #openstack-infra | 09:08 | |
| *** dpawlik has quit IRC | 09:45 | |
| *** dpawlik has joined #openstack-infra | 09:51 | |
| *** dansmith has quit IRC | 09:52 | |
| *** bradm has quit IRC | 09:54 | |
| *** ianw has quit IRC | 09:55 | |
| *** dpawlik has quit IRC | 09:57 | |
| *** dansmith has joined #openstack-infra | 09:58 | |
| *** ianw has joined #openstack-infra | 09:59 | |
| *** betherly has joined #openstack-infra | 10:01 | |
| *** xinranwang has joined #openstack-infra | 10:08 | |
| *** gfidente has joined #openstack-infra | 10:13 | |
| *** ociuhandu has joined #openstack-infra | 10:17 | |
| *** ociuhandu_ has joined #openstack-infra | 10:25 | |
| *** ociuhandu has quit IRC | 10:25 | |
| *** dpawlik has joined #openstack-infra | 10:29 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:31 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** udesale has quit IRC | 10:38 | |
| *** Lucas_Gray has joined #openstack-infra | 10:44 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 10:44 |
| *** bhavikdbavishi has quit IRC | 11:01 | |
| *** tdasilva_ is now known as tdasilva | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:12 | |
| *** yamamoto has quit IRC | 11:18 | |
| *** yamamoto has joined #openstack-infra | 11:19 | |
| *** Lucas_Gray has quit IRC | 11:19 | |
| *** ociuhandu_ has quit IRC | 11:34 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** ociuhandu has joined #openstack-infra | 11:35 | |
| *** ramishra has quit IRC | 11:35 | |
| *** Lucas_Gray has joined #openstack-infra | 11:36 | |
| *** ramishra has joined #openstack-infra | 11:46 | |
| *** jamesmcarthur has joined #openstack-infra | 11:50 | |
| *** dpawlik has quit IRC | 11:52 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** jamesmcarthur has quit IRC | 12:03 | |
| *** jaosorior has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:12 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** jcoufal has joined #openstack-infra | 12:14 | |
| *** dpawlik has joined #openstack-infra | 12:16 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** eharney has joined #openstack-infra | 12:23 | |
| *** psachin has quit IRC | 12:25 | |
| *** ykarel is now known as ykarel|afk | 12:32 | |
| *** ricolin has quit IRC | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:37 | |
| *** ykarel|afk has quit IRC | 12:38 | |
| *** jpena|lunch is now known as jpena | 12:42 | |
| *** jamesmcarthur has joined #openstack-infra | 12:44 | |
| *** xinranwang has quit IRC | 12:47 | |
| *** ociuhandu has joined #openstack-infra | 12:50 | |
| *** jamesmcarthur has quit IRC | 12:51 | |
| jpena | hi! I just noticed the centos AFS mirror has not been updated since 6 days ago (http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1&fullscreen&panelId=14). Is it expected or an issue? | 12:51 |
| *** jaosorior has joined #openstack-infra | 12:54 | |
| *** ykarel|afk has joined #openstack-infra | 13:00 | |
| *** ykarel|afk is now known as ykarel | 13:00 | |
| *** joeguo has quit IRC | 13:01 | |
| *** jamesmcarthur has joined #openstack-infra | 13:01 | |
| *** iurygregory has quit IRC | 13:02 | |
| *** jamesmcarthur has quit IRC | 13:02 | |
| *** jamesmcarthur has joined #openstack-infra | 13:03 | |
| *** jcoufal has quit IRC | 13:04 | |
| *** sthussey has joined #openstack-infra | 13:11 | |
| *** mriedem has joined #openstack-infra | 13:13 | |
| *** n-saito has quit IRC | 13:15 | |
| *** aedc has joined #openstack-infra | 13:16 | |
| *** aaronsheffield has joined #openstack-infra | 13:21 | |
| frickler | jpena: this seems to be an issue with quotas: rsync: close failed on "/afs/.openstack.org/mirror/centos/7/updates/x86_64/Packages/.java-11-openjdk-jmods-11.0.4.11-0.el7_6.i686.rpm.tI3lf0": Disk quota exceeded (122) | 13:22 |
| donnyd | today I think we are just waiting on https://review.opendev.org/#/c/673378/ to be merged. The parts I was waiting on came in yesterday | 13:22 |
| *** goldyfruit has joined #openstack-infra | 13:24 | |
| frickler | infra-root: the above quota issue seems to also block other rsyncs like opensuse, which doesn't seem to have been executed for a couple of days | 13:25 |
| *** jcoufal has joined #openstack-infra | 13:26 | |
| *** iurygregory has joined #openstack-infra | 13:29 | |
| *** smrcascao9 has joined #openstack-infra | 13:31 | |
| smrcascao9 | hi guys | 13:31 |
| smrcascao9 | do you can recomend me some framework or way to test my cluster openstack ? | 13:31 |
| fungi | smrcascao9: you're probably looking for https://refstack.openstack.org/ | 13:33 |
| *** yamamoto has quit IRC | 13:33 | |
| fungi | smrcascao9: if you're looking for something more rigorous than interoperability testing (you haven't said), then you may consider asking in the #openstack-qa channel | 13:34 |
| fungi | they maintain a variety of tools for holistically testing changes to openstack | 13:34 |
| smrcascao9 | hi fungi thanks for your response | 13:34 |
| smrcascao9 | i m looking for same ways to test my infra | 13:34 |
| smrcascao9 | i m using openstack with openshift | 13:35 |
| smrcascao9 | and i need have some way to test them | 13:35 |
| openstackgerrit | Merged openstack/project-config master: set FN to boot from volume for Cinder NVME backend https://review.opendev.org/673378 | 13:36 |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| *** yamamoto has quit IRC | 13:36 | |
| *** yamamoto has joined #openstack-infra | 13:37 | |
| fungi | smrcascao9: you've accidentally found the irc channel where we collaborate on maintaining the developer infrastructure for the openstack community. this is not a channel about building an infrastructure with openstack | 13:39 |
| *** aedc has quit IRC | 13:40 | |
| fungi | we maintain revision control systems, mailing lists, irc bots, wikis, continuous integration services, code review systems and that sort of stuff | 13:41 |
| smrcascao9 | thanks for all fungi and sorry my mistake :) | 13:41 |
| fungi | smrcascao9: see https://wiki.openstack.org/wiki/IRC for a list of openstack irc channels and their uses | 13:41 |
| *** yamamoto has quit IRC | 13:42 | |
| efried | corvus: I think I figured out what gertty was doing with all those sync tasks yesterday... | 13:47 |
| efried | I happened to have some old, stale repos cloned under what gertty thought was my git root. And now, the last-mod stamps on them are current. Which means gertty saw them and, I guess, refreshed them, even the ones I'm not "subscribed to". | 13:48 |
| donnyd | smrcascao9: maybe also give #openstack-mentoring a spin | 13:49 |
| *** tdasilva has quit IRC | 13:49 | |
| fungi | ooh, great idea donnyd | 13:49 |
| donnyd | :) | 13:49 |
| fungi | efried: gertty will also fetch changes in projects you're not subscribed to if you open them directly or follow a depends-on or similar url from another change you're viewing | 13:50 |
| donnyd | So I am measuring the fdowload speeds for glance, and they are borderline painful. 40G network, tons of cores, and nvme local storage.. glance should be able to do more than 98Mb/s in download speeds | 13:50 |
| fungi | efried: that's where a bunch of my "extra" repos end up coming from | 13:50 |
| efried | fungi: Noted, but these were repos under a directory I hadn't touched since... before I installed this computer. (they were copied over from a former disk image) | 13:51 |
| fungi | donnyd: ouch. is that how fast nova is fetching images to warm its cache? | 13:51 |
| fungi | efried: yeah, i find that strange too | 13:51 |
| donnyd | cinder is the backend because I boot from volume. Setup cinder image-cache so I don't have to pay for that slow speed on every instance | 13:52 |
| *** yikun has quit IRC | 13:52 | |
| donnyd | I have load tested up to 30 simultaneous instance launches without the image cached... just to make sure it will work well.. the first instance takes a bit from the slow glance speeds | 13:53 |
| donnyd | but the ones after boot up quickly like they do in ceph | 13:53 |
| donnyd | efried: yea, I can't really figure out why the speeds are so slow on essentially an http server and local files | 13:54 |
| *** factor has joined #openstack-infra | 13:54 | |
| *** tdasilva has joined #openstack-infra | 13:54 | |
| donnyd | I am testing it with a swift backend to see if its any faster tonight | 13:54 |
| fungi | where does glance currently store its images? local disk? | 13:55 |
| fungi | it does seem like there could be some optimization opportunity if glance was storing images in cinder and then cinder was serving those images as the basis for boot volumes | 13:55 |
| donnyd | yea | 13:56 |
| fungi | i have no idea if anything like that is implemented as a feature though | 13:56 |
| donnyd | I am a little tight on space in cinder, so I can't really do that. Have to use a different server... but with local nvme, I can easily outperform a 10G link... and start getting into 40G territory.. the performance I am getting is a lot like 1G territory | 13:57 |
| fungi | jumbo frames on that broadcast domain? | 13:58 |
| fungi | granted, even normal ethernet frame sizes should be capable of more than you're seeing | 13:58 |
| donnyd | No | 13:59 |
| donnyd | I only use jumbo for iscsi, i don't want to deal with mtu games | 13:59 |
| donnyd | we would be talking maybe an extra 50Mb/s with jumbos anyways | 13:59 |
| donnyd | yea, I would surely think so | 13:59 |
| fungi | you ought to at least be able to do close to 1gbps with a single flow at normal mtu sizes | 13:59 |
| donnyd | That is what I am thinking | 14:00 |
| fungi | granted if you want to push beyond 1gbps you either need larger frames or additional flows | 14:00 |
| fungi | but you're still an order of magnitude from there, sounds like | 14:00 |
| donnyd | single threaded read on local disk | 14:01 |
| fungi | has some link gone back in time and autonegotiated 100mbps like it's the 90s all over again? | 14:01 |
| donnyd | Run status group 0 (all jobs): | 14:01 |
| donnyd | READ: bw=2558MiB/s (2683MB/s), 2558MiB/s-2558MiB/s (2683MB/s-2683MB/s), io=4096MiB (4295MB), run=1601-1601msec | 14:01 |
| donnyd | LOL | 14:01 |
| donnyd | so its surely not a local disk read issue | 14:01 |
| donnyd | even tiny little 4k blocks i see 4x more in performance | 14:02 |
| donnyd | READ: bw=405MiB/s (425MB/s), 405MiB/s-405MiB/s (425MB/s-425MB/s), io=4096MiB (4295MB), run=10104-10104msec | 14:02 |
| *** rpittau is now known as rpittau|afk | 14:02 | |
| fungi | and you're sure the transfers are only managing 100 megabits per second, not 100 megabytes per second? | 14:02 |
| donnyd | So there will be a bit more tinkering to get glance tuned up and running well | 14:03 |
| fungi | the latter isn't too far off from the theoretical limit of normal frame sizes through a single flow | 14:03 |
| *** e0ne has quit IRC | 14:04 | |
| fungi | so if it was 100 megabytes per second i'd say you were in the ballpark of what to expect | 14:04 |
| donnyd | well the other http servers I have on my network have no issues with getting at least half of wireline for 10G | 14:04 |
| fungi | (unless glance parallelizes transfers) | 14:04 |
| donnyd | which is a lot more like 500Mb/s | 14:04 |
| efried | fungi: How does a guy expand/see inline comments in gertty? | 14:06 |
| donnyd | single thread network performance from cinder volume server to glance | 14:06 |
| donnyd | [ 5] 0.00-1.00 sec 1.48 GBytes 12.7 Gbits/sec 88 1.21 MBytes | 14:06 |
| donnyd | [ 5] 1.00-2.00 sec 2.45 GBytes 21.0 Gbits/sec 0 1.50 MBytes | 14:06 |
| donnyd | [ 5] 2.00-3.00 sec 2.54 GBytes 21.8 Gbits/sec 17 1.47 MBytes | 14:06 |
| donnyd | [ 5] 3.00-4.00 sec 1.41 GBytes 12.1 Gbits/sec 13 1.50 MBytes | 14:06 |
| donnyd | [ 5] 4.00-5.00 sec 2.88 GBytes 24.8 Gbits/sec 71 1.14 MBytes | 14:06 |
| donnyd | [ 5] 5.00-6.00 sec 3.09 GBytes 26.5 Gbits/sec 128 851 KBytes | 14:06 |
| donnyd | [ 5] 6.00-7.00 sec 3.20 GBytes 27.5 Gbits/sec 2 1.68 MBytes | 14:06 |
| donnyd | [ 5] 7.00-8.00 sec 2.74 GBytes 23.6 Gbits/sec 95 1.34 MBytes | 14:06 |
| donnyd | [ 5] 8.00-9.00 sec 3.04 GBytes 26.2 Gbits/sec 108 1.25 MBytes | 14:06 |
| donnyd | [ 5] 9.00-10.00 sec 3.04 GBytes 26.1 Gbits/sec 60 1.33 MBytes | 14:06 |
| donnyd | - - - - - - - - - - - - - - - - - - - - - - - - - | 14:06 |
| donnyd | [ ID] Interval Transfer Bitrate Retr | 14:06 |
| donnyd | 2019-07-30 10:06:18.643 2346 INFO cinder.image.image_utils [req-fa0cacd1-f9ba-4da2-be18-b2396a05d397 4970ad73e2554071a348e168b3e8ebe4 44651bdb0d7a4d28adecd7653d39a38c - - -] Image download 6567.94 MB at 101.05 MB/s | 14:07 |
| donnyd | 2019-07-30 10:07:00.724 2346 INFO cinder.image.image_utils [req-fa0cacd1-f9ba-4da2-be18-b2396a05d397 4970ad73e2554071a348e168b3e8ebe4 44651bdb0d7a4d28adecd7653d39a38c - - -] Converted 16118.62 MB image at 405.34 MB/s | 14:07 |
| fungi | efried: same way a girl does... they just show up expanded by default when looking at the diff i think? granted i have unified diff turned on rather than side-by-side so i don't know if that influences it | 14:08 |
| * fungi didn't realize gertty was gender aware, but will double-check the source code ;) | 14:09 | |
| efried | fungi: I have whatever the default diff is, and I'm not seeing them in the diff. /me checks again... | 14:09 |
| efried | yeah, apparently so. | 14:09 |
| AJaeger | fungi: tell us once you find the magic config option ;) | 14:10 |
| efried | oh, I'm looking at the latest PS, but the comment I was looking for is in an earlier PS. | 14:10 |
| fungi | efried: you might be looking at an old revision of a change if it hasn't synced yet. ctrl-r in the change view will prioritize syncing that change | 14:11 |
| fungi | efried: oh, if you want to see inline comments for an earlier patchset, you can either select the diff option for that specific patchset in the change view, or you can hit the p key (by default if you haven't remapped it) in the diff view to select start and end patchsets for the diff | 14:12 |
| efried | fungi: When I made the earlier PS show up in the diff, the comment appeared. | 14:12 |
| efried | not sure how to save my reply though | 14:17 |
| fungi | efried: if you hit enter on a line you can just start typing and when you exit it will be saved with your review (when you use the reply button on the change view or whatever) | 14:18 |
| *** ykarel is now known as ykarel|away | 14:19 | |
| efried | fungi: "exit" like Esc? | 14:19 |
| fungi | esc or whatever you've bound to exit the diff view, yes | 14:19 |
| efried | gertty semes to be remembering my comment, but when I did a reply on the review, it didn't get pushed. | 14:19 |
| *** yamamoto has joined #openstack-infra | 14:19 | |
| fungi | was the comment associated with a different patchset than the current patchset? | 14:19 |
| efried | the comment was on a non-latest patch set, which I'm assuming... yes | 14:19 |
| fungi | you need to reply to that patchset then | 14:20 |
| fungi | same as in the gerrit webui, i believe | 14:20 |
| fungi | donnyd: definitely looks like something with glance is suboptimal, i agree. it it going through apache mod_wsgi? | 14:21 |
| efried | fungi: no, in webui your change gets saved regardless. If you're on PS4 and you save a PS3 comment, it shows up tagged with 'PS3' | 14:21 |
| efried | but it always saves | 14:21 |
| fungi | donnyd: not that i have the first clue how to tune that | 14:21 |
| efried | trying to figure out how to "be" on an earlier ps from the change set view... | 14:21 |
| fungi | either select the diff button for the patchset you want to comment on or use 'p' to set that patchset as the current one (right column) in the diff view | 14:22 |
| donnyd | fungi: Its not right now, but I am surely going to test all the options to see what gives the best performance | 14:22 |
| fungi | efried: you probably need to expand that patchset in the change view to see the buttons for it (move your cursor line over it and press return). and then use the review button for that patchset (exposed the same way) to save comments for it | 14:23 |
| fungi | efried: where it says "Patch Set 2 abcdefsomehash..." or whatever in the change view | 14:24 |
| fungi | highlight that line and hit enter and you get its file list and buttons like < Review > < Diff > < Local Checkout > < Local Cherry-Pick > | 14:25 |
| fungi | efried: it's set up basically like the "old change screen" in gerrit 2.8 and before, but if you never used older versions of gerrit it may not be as intuitive i guess | 14:25 |
| fungi | (the "new" change screen we have in the 2.13 webui is terrible in my opinion) | 14:26 |
| efried | fungi: Okay, at least one of my attempts therein seems to have worked. | 14:26 |
| *** michael-beaver has joined #openstack-infra | 14:27 | |
| *** sreejithp has joined #openstack-infra | 14:27 | |
| efried | I never used "old", and I have no complaints about "new". What wasn't intuitive here was that, when I punched Review, nothing showed up in the box. As in the web UI, it would be nice if my inline comments were displayed (read-only) so I know they're getting committed. | 14:27 |
| efried | and also as in the webui, it would be nice if all my inline comments were pushed, regardless of which ps I left them on and which ps I'm technically "responding" to. | 14:28 |
| *** yamamoto has quit IRC | 14:28 | |
| efried | I guess I should start storyboarding these things... | 14:29 |
| *** Lucas_Gray has quit IRC | 14:29 | |
| *** Lucas_Gray has joined #openstack-infra | 14:32 | |
| *** Lucas_Gray has quit IRC | 14:32 | |
| smcginnis | donnyd: It looks like the single threadedness comes down to the way the glanceclient downloads and uploads images. | 14:33 |
| *** e0ne has joined #openstack-infra | 14:33 | |
| smcginnis | donnyd: It ends up using the requests library to make those calls, and they are single calls per image from what I could glean. | 14:33 |
| *** dpawlik has quit IRC | 14:34 | |
| smcginnis | I don't believe there is anything built in to requests to try to multithread a download. | 14:34 |
| AJaeger | infra-root, tc-as-a-service repo was imported but never used, cores not even setup. I'm in contact of one of the authors and they say it's dead. I'll go through the usual retirement process - but could you add me to https://review.opendev.org/#/admin/groups/1769,members , please? The list is currently empty. | 14:35 |
| mordred | AJaeger: done | 14:36 |
| AJaeger | thanks, mordred | 14:36 |
| fungi | he's faster than i am | 14:36 |
| clarkb | tc-as-a-service, that is what fungi does for us right? | 14:36 |
| *** bobh has joined #openstack-infra | 14:36 | |
| mordred | smcginnis: we multithread image uploads in sdk - if the upload is going to swift | 14:37 |
| AJaeger | ;) | 14:37 |
| mordred | if there is a way to upload chunks in paralell that would be supported by glance server-side, we've got the things in place to make use of it sdk-sie | 14:37 |
| mordred | side | 14:38 |
| elod | hi infra team! I have a question regarding excluding a job: in murano (stable/ocata), the following job runs on my patch: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L688 . However there isn't any install-guide in ocata branch of murano. Is there a way to exclude this job? Somehow in .zuul.yaml of the branch/project? | 14:38 |
| smcginnis | mordred: Is that using the glance client, or was this part of the direct implementation in the SDK? | 14:38 |
| mordred | smcginnis: direct. we don't use any *client libs | 14:39 |
| *** factor has quit IRC | 14:39 | |
| *** factor has joined #openstack-infra | 14:39 | |
| smcginnis | mordred: But yeah, I don't think there is any support on the glance side to do chunked based uploads that would allow splitting up an image and pushing in parallel. | 14:39 |
| mordred | yeah. that would be the first bit | 14:39 |
| *** Lucas_Gray has joined #openstack-infra | 14:39 | |
| *** pgaxatte has quit IRC | 14:39 | |
| mordred | however - should someone implement that, we'd happily consume it :) | 14:39 |
| smcginnis | I'm kind of suprised no one has pushed that yet. | 14:40 |
| mordred | it is one of the (very few) nice parts about the "upload to swift then have glance import from swift" workflow | 14:40 |
| donnyd | wsgi is the same result | 14:41 |
| mordred | the rest of that workflow blows - but the "upload to swift" part works real well because of the parallel chunk uploading | 14:41 |
| donnyd | maybe a swift backend will do parallel downloading too | 14:41 |
| clarkb | elod: can you link to an example job (that gives us information on where it is defined for that project) | 14:42 |
| elod | clarkb: this is the job: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L688 | 14:42 |
| clarkb | sorry can you link to the logs of an example build of that job? | 14:43 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Start retiring tc-as-a-service https://review.opendev.org/673543 | 14:43 |
| AJaeger | and here'S step one for retiring the repo ^ | 14:43 |
| elod | clarkb: and this is the project's zuul.yaml: https://review.opendev.org/#/c/668181/1/.zuul.yaml | 14:43 |
| clarkb | the inventory of the job (whcih we log) contains details on why zuul decided to run that job. From that we can figure out the best way to exclude it | 14:43 |
| *** icarusfactor has joined #openstack-infra | 14:43 | |
| elod | clarkb: and the logs: http://logs.openstack.org/81/668181/1/check/build-openstack-install-guide/c90482a/ | 14:44 |
| AJaeger | elod: do you have isntall-guide on any branch? It'S in project-config set up for your repo... | 14:44 |
| elod | AJaeger: in pike there was, but after a while it was refactored as I saw | 14:44 |
| clarkb | it does not come from publish-openstack-docs-pti | 14:45 |
| AJaeger | elod: so, one way: Remove the job completely - and neither publish on pike or ocata anymore... | 14:45 |
| AJaeger | clarkb: install-guide-jobs in project-config | 14:45 |
| clarkb | http://logs.openstack.org/81/668181/1/check/build-openstack-install-guide/c90482a/zuul-info/inventory.yaml is the log file I'm looking for | 14:45 |
| clarkb | AJaeger: thanks | 14:45 |
| *** factor has quit IRC | 14:46 | |
| *** jcoufal has quit IRC | 14:46 | |
| AJaeger | I assume nobody will touch it in pike either, so removing the template is the quickest approach. But if you want to run it on pike, it needs a bit more work... | 14:46 |
| AJaeger | Sorry, need to step out for a bit... | 14:46 |
| clarkb | smcginnis: elod fyi https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L77-L103 that is what publish-openstack-docs-pti gets you | 14:47 |
| clarkb | elod: as AJaeger points out it comes from https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L2801 | 14:47 |
| clarkb | elod: you can remove that line as AJaeger suggests assuming you don't need it for pike. If you do want it for pike you'll need to remove the template then add the jobs back with a branch match for only pike. You can do that in the pipeline definition I just linked too | 14:48 |
| *** icarusfactor has quit IRC | 14:50 | |
| *** icarusfactor has joined #openstack-infra | 14:50 | |
| clarkb | donnyd: looks like most of our images have been uploaded to fn at this point (debian buster and opensuse 423 seem to have failed?) | 14:50 |
| elod | clarkb: thanks, looking | 14:50 |
| donnyd | clarkb: if they are trying in the last 30 minutes or so I am moving to wsgi to see if performance is any better | 14:51 |
| *** bhavikdbavishi has joined #openstack-infra | 14:52 | |
| *** dklyle has quit IRC | 14:52 | |
| clarkb | donnyd: Ithink they are failing because we are failing to build newer images for those image types | 14:52 |
| clarkb | I'll have to look at build logs when meetings are complete | 14:53 |
| *** dklyle has joined #openstack-infra | 14:53 | |
| *** takamatsu has joined #openstack-infra | 14:54 | |
| *** e0ne has quit IRC | 14:54 | |
| *** icarusfactor has quit IRC | 14:55 | |
| *** factor has joined #openstack-infra | 14:56 | |
| *** bhavikdbavishi has quit IRC | 14:56 | |
| *** bdodd has joined #openstack-infra | 14:58 | |
| clarkb | 2019-07-30 15:00:41.879 | E: Failed to fetch http://mirror.dfw.rax.openstack.org/debian/dists/buster-backports/main/binary-amd64/Packages 404 Not Found [IP: 2001:4800:7817:103:be76:4eff:fe04:652a 80] | 15:01 |
| clarkb | I'm guessing backports did not exist when we set that up but now does exist? | 15:02 |
| clarkb | but also there was notes of rsyncs failing due to disk space? | 15:02 |
| *** e0ne has joined #openstack-infra | 15:04 | |
| clarkb | I think we are configured to pull backports for buster but they don't exist on our mirror. Will need to dig more after meetings | 15:04 |
| *** hamzy has quit IRC | 15:12 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:14 | |
| *** ricolin has joined #openstack-infra | 15:14 | |
| clarkb | fungi: http://cacti.openstack.org/cacti/graph_view.php you can see the git gc :) | 15:28 |
| *** chandankumar is now known as raukadah | 15:29 | |
| *** eernst has joined #openstack-infra | 15:29 | |
| *** Goneri has quit IRC | 15:30 | |
| *** kopecmartin is now known as kopecmartin|off | 15:30 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 15:32 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 15:32 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 15:32 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 15:32 |
| AJaeger | config-core, could you review https://review.opendev.org/673543 , please? | 15:32 |
| *** mriedem has quit IRC | 15:33 | |
| AJaeger | thanks, mnaser and clarkb ! | 15:34 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring tc-as-a-service https://review.opendev.org/673563 | 15:35 |
| *** mriedem has joined #openstack-infra | 15:35 | |
| donnyd | clarkb: I am going to open up the tap a little and watch what it does. the glance thing is what it is. I will work towards a faster fix, but for now it's not going to be too much different from the other providers | 15:35 |
| AJaeger | config-core, a simple job rename for grafana, please review https://review.opendev.org/672290 | 15:36 |
| fungi | donnyd: yeah, we're used to seeing a brief spike in boot times immediately after images are uploaded | 15:36 |
| clarkb | donnyd: ok | 15:36 |
| clarkb | donnyd: our popular iamges are all in fn now so you should have plenty of workload happy to run there | 15:36 |
| snierodz | AJaeger: we've put up a few patches since the 'Depends-On' link error and we are getting closer to fixing the gate but we're currently running into a MYSQL error in our functional tests. Link:http://logs.openstack.org/25/672725/12/check/neutron-classifier-functional-dsvm/1ca6378/job-output.txt.gz#_2019-07-30_13_08_37_890742 | 15:37 |
| *** ykarel|away has quit IRC | 15:38 | |
| snierodz | The issue is to do with mysql being preinstalled and its failling to install and update the password as a result | 15:38 |
| snierodz | Would you have any tips in resolving this type of error? | 15:39 |
| *** whoami-rajat has quit IRC | 15:39 | |
| AJaeger | snierodz: better ask the QA team on #openstack-qa, I have no direct idea | 15:39 |
| snierodz | AJaeger: no worries, thanks | 15:40 |
| donnyd | uggg... so something is wrong with the networking... because when nodepool launches an instance its only getting a v6 address | 15:40 |
| *** ekultails has joined #openstack-infra | 15:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix typo in download-artifact https://review.opendev.org/673566 | 15:46 |
| openstackgerrit | Merged openstack/project-config master: Start retiring tc-as-a-service https://review.opendev.org/673543 | 15:47 |
| donnyd | Any ideas on why it would be ? | 15:47 |
| donnyd | https://www.irccloud.com/pastebin/fx4LwDB2/ | 15:47 |
| clarkb | donnyd: is there a default ipv4 subnet/network? | 15:48 |
| donnyd | So when i launch an instance on my end using the network flag, it grabs ip addresses from both... but when nodepool does it, its only getting v6 | 15:48 |
| clarkb | and by default I think neutron uses the only one that is present otherwise you have to specifiy | 15:48 |
| clarkb | mordred: ^ may also know since that likely involves openstacksdk and client config | 15:48 |
| donnyd | its a dual stack network just like before | 15:48 |
| mordred | is nodepool configured with the network flag too? | 15:50 |
| openstackgerrit | Merged zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 15:50 |
| clarkb | mordred: it shouldn't be | 15:50 |
| clarkb | should rely on the defaults of pick the only network available I think | 15:51 |
| mordred | if there are more than one available network that a vm can attach to and no other config or distinguishing features, sdk will pick the first one | 15:51 |
| openstackgerrit | Merged zuul/zuul-jobs master: Update testing section https://review.opendev.org/672820 | 15:51 |
| mordred | oh - that is - v6_test should be the network, right? | 15:51 |
| mordred | and that's the one it's using and it's getting the v6 address? | 15:51 |
| mordred | when you say "nodepool is only getting the v6" ... what does that mean? | 15:52 |
| *** gyee has joined #openstack-infra | 15:52 | |
| clarkb | mordred: https://www.irccloud.com/pastebin/fx4LwDB2/ the instances do not get ipv4 addresses from neutron, only ipv6 addresses | 15:52 |
| clarkb | they should get a global routable ipv6 addr and a private 10 net ipv4 addr that is NATed when talking to github | 15:53 |
| clarkb | mordred: the v6_test instance got both | 15:53 |
| mordred | oh - I'm reading that paste better now | 15:53 |
| mordred | it's the openstackzuul network that is providing both v4 and v6? | 15:53 |
| clarkb | looks like it yes | 15:54 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 15:54 |
| mordred | ok. then that's weird - there shouldn't be much sdk can do related to that operation. it's going to do create_server(network='openstackzuul') *handwave* - and if openstackzuul is configured to give both addresses that's what will happen | 15:54 |
| clarkb | donnyd: is the ipv4 ip range/dhcp pool big neough to hand out more than the one addr? | 15:55 |
| fungi | donnyd: when you say "when i launch an instance on my end using the network flag" are you using openstackclient/openstacksdk? or something else? | 15:55 |
| donnyd | osc | 15:55 |
| mordred | is it possible that the v6_test instance was created by a different user on this project and that user had a different set of permissions, and that possibly the subnet on the network isn't usable by the nodepool user? | 15:55 |
| donnyd | openstack server create --image fedora-30 --flavor 8cpu-8GBram-80GBdisk --key-name precision --network openstackzuul v6_test | 15:55 |
| mordred | clarkb: that too | 15:55 |
| donnyd | that is possible | 15:56 |
| * mordred LOVES that a server boot requests a network, but the qualities a user cares about are the subnet objects, which cannot be selected at boot | 15:56 | |
| mordred | LOVES that | 15:57 |
| clarkb | mordred: that is because neutron decided to go with a literal network abstraction | 15:57 |
| clarkb | #status log Increased AFS quota for /afs/.openstack.org/mirror/ubuntu and /afs/.openstack.org/mirror/centos by 50GB each. | 15:57 |
| openstackstatus | clarkb: finished logging | 15:57 |
| clarkb | the RO/backup volumes don't reflect that increase yet but I think they will after their next vos release | 15:57 |
| clarkb | I'm going to get bike ride out of the way now, but then will be back to dig into fn networking and debian buster mirroring as necessary when I return | 15:58 |
| clarkb | fungi: ^ you might have ideas on the buster backports mirroring? we seem to have plenty of quota for it | 15:58 |
| clarkb | and we don't exclude it from what I see | 15:58 |
| *** e0ne has quit IRC | 15:59 | |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| donnyd | mordred: I think you nailed it | 16:00 |
| donnyd | openstack subnet list | 16:00 |
| donnyd | +--------------------------------------+------------------+--------------------------------------+-------------------------+ | 16:00 |
| donnyd | | ID | Name | Network | Subnet | | 16:00 |
| donnyd | +--------------------------------------+------------------+--------------------------------------+-------------------------+ | 16:00 |
| donnyd | | d4950399-ead2-430a-a140-bb00aa994439 | openstackzuul_v6 | 3b5f94f2-8fd7-4626-b90e-c602ccbf9320 | 2001:470:e045:8000::/64 | | 16:00 |
| donnyd | +--------------------------------------+------------------+--------------------------------------+-------------------------+ | 16:00 |
| donnyd | changed myself to a regular member and i cannot see the subnet | 16:00 |
| donnyd | easy fix | 16:00 |
| donnyd | thanks | 16:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix typo in download-artifact https://review.opendev.org/673566 | 16:01 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Remove old install-guide-jobs from murano https://review.opendev.org/673574 | 16:01 |
| donnyd | https://www.irccloud.com/pastebin/wJfy78EV/ | 16:03 |
| mordred | \o/ | 16:03 |
| donnyd | Yea, now it should be good to go. Thank you mordred :) | 16:03 |
| mordred | donnyd: I think this means I know too much about the api structure | 16:03 |
| fungi | clarkb: i think there is no actual backports suite for buster yet? i'll double-check | 16:04 |
| donnyd | what i don't understand is the v6 network was created the exact same way at the v4 netwokr | 16:04 |
| donnyd | I have my finger crossed that performance is as good in testing as it is in real life | 16:06 |
| donnyd | iops per instance are up 4X | 16:06 |
| elod | clarkb AJaeger : can you please review my patch? I looked at the files you showed me and... Is my thinking correct, that I can safely remove install-guide-jobs for murano? https://review.opendev.org/673574 | 16:06 |
| donnyd | under load before 10K.. now 50k(ish) | 16:06 |
| *** whoami-rajat has joined #openstack-infra | 16:07 | |
| AJaeger | elod: LGTM | 16:07 |
| donnyd | what does the "ready nodes" metric represent in grafana? | 16:08 |
| Shrews | donnyd: nodes that are booted and ready to be assigned to a node request from zuul | 16:08 |
| donnyd | IC | 16:08 |
| fungi | clarkb: looks like buster-backports does have contents on debian's mirrors now. i'll take a closer look at the log | 16:09 |
| elod | AJaeger: thanks! | 16:09 |
| *** panda has quit IRC | 16:09 | |
| *** lpetrut has quit IRC | 16:11 | |
| *** panda has joined #openstack-infra | 16:13 | |
| *** Lucas_Gray has quit IRC | 16:14 | |
| fungi | clarkb: looks to be a missing/incorrect key reprepro is trying to check for a signature from: | 16:18 |
| fungi | ERROR: Condition '2B90D010' not fullfilled for '/afs/.openstack.org/mirror/debian/lists/debian_buster_Release.gpg'. | 16:18 |
| openstackgerrit | Merged zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 16:18 |
| fungi | Error: Not enough signatures found for remote repository debian (http://ftp.us.debian.org/debian buster)! | 16:18 |
| *** ykarel|away has joined #openstack-infra | 16:19 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-jobs master: Add auth config to kubelet user for buildset registries https://review.opendev.org/673351 | 16:19 |
| *** lucasagomes has quit IRC | 16:20 | |
| *** lseki has joined #openstack-infra | 16:21 | |
| *** sparkycollier has quit IRC | 16:23 | |
| *** evgenyl has quit IRC | 16:24 | |
| AJaeger | config-core, https://review.opendev.org/673563 finishes retiring of tc-as-a-service. Could I get review-as-a-service, please? | 16:24 |
| AJaeger | bbl | 16:24 |
| *** JpMaxMan has quit IRC | 16:24 | |
| *** tdasilva_ has joined #openstack-infra | 16:24 | |
| *** sparkycollier has joined #openstack-infra | 16:25 | |
| *** ginopc has quit IRC | 16:26 | |
| *** JpMaxMan has joined #openstack-infra | 16:26 | |
| *** evgenyl has joined #openstack-infra | 16:26 | |
| *** tdasilva has quit IRC | 16:27 | |
| *** rajinir has quit IRC | 16:28 | |
| *** rajinir has joined #openstack-infra | 16:30 | |
| fungi | clarkb: i think i found where to solve it, just trying to figure out what keys we should be including there | 16:30 |
| *** factor has quit IRC | 16:31 | |
| *** factor has joined #openstack-infra | 16:31 | |
| *** trident has quit IRC | 16:32 | |
| *** igordc has joined #openstack-infra | 16:33 | |
| *** gfidente has quit IRC | 16:33 | |
| *** Goneri has joined #openstack-infra | 16:33 | |
| *** JpMaxMan has quit IRC | 16:33 | |
| *** evgenyl has quit IRC | 16:34 | |
| *** pcrews has quit IRC | 16:35 | |
| *** evgenyl has joined #openstack-infra | 16:36 | |
| *** rajinir has quit IRC | 16:36 | |
| *** JpMaxMan has joined #openstack-infra | 16:37 | |
| *** trident has joined #openstack-infra | 16:37 | |
| *** rajinir has joined #openstack-infra | 16:38 | |
| *** jpena is now known as jpena|off | 16:38 | |
| *** sgw has quit IRC | 16:42 | |
| *** JpMaxMan_ has joined #openstack-infra | 16:46 | |
| *** ociuhandu has quit IRC | 16:46 | |
| *** ociuhandu has joined #openstack-infra | 16:47 | |
| *** JpMaxMan has quit IRC | 16:47 | |
| *** JpMaxMan_ is now known as JpMaxMan | 16:47 | |
| *** sparkycollier_ has joined #openstack-infra | 16:47 | |
| *** sparkycollier has quit IRC | 16:47 | |
| *** sparkycollier_ is now known as sparkycollier | 16:47 | |
| *** jcoufal has joined #openstack-infra | 16:48 | |
| *** Goneri has quit IRC | 16:50 | |
| donnyd | fungi: or clarkb Can I add a second interface to the mirror node. It would seem something doesn't work correctly with routing traffic on the v6 side | 16:52 |
| *** jcoufal_ has joined #openstack-infra | 16:53 | |
| *** goldyfruit has quit IRC | 16:53 | |
| donnyd | http://logs.openstack.org/53/673553/2/check/openstack-tox-py27/eb478d4/job-output.txt#_2019-07-30_15_58_50_934464 | 16:54 |
| *** bhavikdbavishi has quit IRC | 16:54 | |
| donnyd | brb | 16:54 |
| *** jcoufal has quit IRC | 16:56 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:56 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:58 | |
| *** jamesmcarthur has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:01 | |
| *** Goneri has joined #openstack-infra | 17:03 | |
| *** markvoelker has quit IRC | 17:03 | |
| *** bhavikdbavishi has quit IRC | 17:07 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:07 | |
| *** ociuhandu has quit IRC | 17:07 | |
| *** ricolin has quit IRC | 17:11 | |
| *** bhavikdbavishi has quit IRC | 17:17 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:18 | |
| *** jcoufal_ has quit IRC | 17:22 | |
| *** ralonsoh has quit IRC | 17:23 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add archive signing key for Debian 10/buster https://review.opendev.org/673584 | 17:25 |
| fungi | clarkb: ^ | 17:25 |
| clarkb | donnyd: that should be ok and udev may even auto config ure it | 17:25 |
| clarkb | fungi: thanks | 17:25 |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| clarkb | fungi: why switch from Release and Release.gpg to InRelease? | 17:29 |
| clarkb | Both should work, just curious if something important changed there | 17:30 |
| mordred | clarkb, fungi: if you're bored, could I get reviews on https://review.opendev.org/#/c/672273/ https://review.opendev.org/#/c/673141 and https://review.opendev.org/#/c/673142 ? | 17:30 |
| fungi | clarkb: https://lists.debian.org/debian-devel/2019/07/msg00153.html | 17:33 |
| clarkb | gotcha way of the future and all that | 17:34 |
| clarkb | mordred: I'm not sure bored is the right word but I'll take a look | 17:34 |
| *** tesseract has quit IRC | 17:34 | |
| clarkb | centos read only quota has updated | 17:35 |
| clarkb | and it appears to be using more that it had before so I'm guessing that implies a successful update and vos release | 17:35 |
| clarkb | frickler: ^ thank you for catching that | 17:35 |
| fungi | clarkb: to be fair, the timeline proposed by jak was a bit aggressive, but best we switch now rather than be caught by it in the future (we can extend this to the ubuntu repos as well if we see it work out fine) | 17:36 |
| *** bhavikdbavishi1 has joined #openstack-infra | 17:36 | |
| *** bhavikdbavishi has quit IRC | 17:37 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 17:37 | |
| mordred | ++ | 17:37 |
| openstackgerrit | Merged openstack/project-config master: Remove old install-guide-jobs from murano https://review.opendev.org/673574 | 17:38 |
| *** armax has quit IRC | 17:40 | |
| *** bhavikdbavishi has quit IRC | 17:41 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add archive signing key for Debian 10/buster https://review.opendev.org/673584 | 17:41 |
| fungi | clarkb: mordred: ^ i was wrong, too soon to switch to inrelease files | 17:41 |
| fungi | i should have double-checked that the stretch archives included them but they did not | 17:42 |
| mordred | fungi: do | 17:42 |
| fungi | so there's the same change with the old style detached signatures still used | 17:42 |
| clarkb | mordred: dockerfile change +A'd | 17:43 |
| *** tosky has quit IRC | 17:43 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:43 | |
| mordred | clarkb: \o/ | 17:43 |
| clarkb | er changes I got all 3 | 17:44 |
| mordred | yeah. thanks! I'm sure you enjoyed those as much as I enjoyed writing them | 17:44 |
| *** georgk has quit IRC | 17:55 | |
| *** fdegir has quit IRC | 17:55 | |
| *** fdegir has joined #openstack-infra | 17:56 | |
| *** georgk has joined #openstack-infra | 17:56 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Cleanup extra /'s in manifest render view https://review.opendev.org/673605 | 18:01 |
| *** priteau has quit IRC | 18:02 | |
| *** jcoufal has joined #openstack-infra | 18:07 | |
| *** jcoufal has quit IRC | 18:07 | |
| *** jcoufal has joined #openstack-infra | 18:07 | |
| donnyd | clarkb it would seem not | 18:10 |
| *** hamzy has joined #openstack-infra | 18:11 | |
| clarkb | donnyd: seems I can't hit the mirror via ipv4? | 18:17 |
| donnyd | Should be back up in a few | 18:17 |
| donnyd | just tinkering with the network to get it working correct on both | 18:18 |
| *** armax has joined #openstack-infra | 18:18 | |
| donnyd | looks like its good on v6, just getting the v4 part sorted | 18:18 |
| *** ykarel|away has quit IRC | 18:22 | |
| donnyd | ok, should be good to go now on both | 18:27 |
| *** yamamoto has joined #openstack-infra | 18:27 | |
| openstackgerrit | Merged opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 18:29 |
| *** yamamoto has quit IRC | 18:31 | |
| *** harlowja has quit IRC | 18:33 | |
| *** jamesmcarthur has quit IRC | 18:35 | |
| clarkb | ubuntu RO mirror path now reflects the higher quota | 18:48 |
| clarkb | donnyd: I'm still getting no route to host via ssh -4 | 18:48 |
| donnyd | yea is still acing wierd | 18:51 |
| *** bobh has quit IRC | 18:56 | |
| donnyd | can you ssh in via the v6 address and check to see what is going on in the instance | 19:00 |
| clarkb | I've got to run the weekly meeting now, but can do that after | 19:01 |
| donnyd | ok cool | 19:01 |
| *** diablo_rojo has joined #openstack-infra | 19:11 | |
| smcginnis | gitea down? | 19:13 |
| fungi | i hope not? | 19:14 |
| fungi | checking | 19:14 |
| fungi | https://opendev.org/ loads for me | 19:14 |
| *** eharney has quit IRC | 19:15 | |
| fungi | https://opendev.org/opendev/system-config/ loading for me too | 19:15 |
| donnyd | fungi: mordred figured out why were we only getting a v6 address this weekend from the mirror | 19:16 |
| fungi | donnyd: i saw! | 19:17 |
| fungi | that guy has a weird v6th sense | 19:17 |
| fungi | smcginnis: was it temporary? or what specifically are you seeing? | 19:17 |
| *** mriedem has quit IRC | 19:20 | |
| smcginnis | fungi: Must have been temporary. I tried to pull up https://opendev.org/openstack/cinder and got a 500 error page. | 19:21 |
| smcginnis | Ironically at the same point in the meeting folks were talking about how much more stable things should be now. :) | 19:22 |
| fungi | yeah, after the meeting i can scour the logs. we've been seeing some rootfs timeouts (independent of the fixes we've put in place) which could be at issue | 19:26 |
| smcginnis | Looks happy enough now. Though I've been noticing a lot of slowness. | 19:27 |
| fungi | also the haproxy currently only does a layer 4 tcp port check of the backends, we could step that up | 19:27 |
| clarkb | smcginnis: we've tracked down the slowness to gitea scanning repos to find when files and dirs were last updated but not caching that data. We made a call for volunteers on the openstack-discuss list when sean mooney pointed it out as we've been focused on reliability over performance currently | 19:28 |
| fungi | assuming the slowness you observe is in browsing file lists of large repositories | 19:28 |
| smcginnis | clarkb: Is the call for volunteers to debug the issue in the gitea source? | 19:29 |
| clarkb | smcginnis: yes | 19:29 |
| clarkb | corvus: and I posted a bunch of details on how you can get started doing that if interested (to the same thread) | 19:30 |
| *** mriedem has joined #openstack-infra | 19:32 | |
| smcginnis | If I find some time that I can devote to that, I'll take a look. Unfortunately pretty low on my priority list at the moment though. | 19:32 |
| fungi | if you happen to know anyone who's interested in hacking on it, please don't hesitate to hook us up | 19:33 |
| *** bobh has joined #openstack-infra | 19:36 | |
| smcginnis | +1 | 19:36 |
| *** bhavikdbavishi has quit IRC | 19:39 | |
| openstackgerrit | Merged opendev/system-config master: Add archive signing key for Debian 10/buster https://review.opendev.org/673584 | 19:42 |
| *** bobh has quit IRC | 19:42 | |
| *** joeguo has joined #openstack-infra | 19:42 | |
| fungi | i'll check the mirror-builder once that ^ gets applied and see if the issues with buster-backports are fixed | 19:43 |
| *** whoami-rajat has quit IRC | 19:57 | |
| clarkb | donnyd: http://paste.openstack.org/show/755135/ ipv4 address seems configured and the routes don't look wrong to me | 19:57 |
| clarkb | however you have the actual details to double check for us | 19:57 |
| donnyd | that should be a private address | 19:58 |
| clarkb | I think fungi was trying to direct attach the ip? | 19:58 |
| clarkb | this was the whole conversation with mordred right? where you don't have to do a private addr to fip and can just direct attach the fip addr? | 19:58 |
| clarkb | fungi: mordred ^ you may want to look at that too | 19:58 |
| donnyd | I can got back to that, just need someone to add in the other interface. For whatever reason the bgp-dragent doesn't want to... it won't let me dual stack on the wan side | 19:59 |
| donnyd | I can do both | 19:59 |
| donnyd | And we this weekend had just set up direct attach to the wan side, so no fip | 19:59 |
| donnyd | Which I am happy to do if someone can just make sure the interfaces in the vm are straight | 20:00 |
| fungi | clarkb: yeah, for whatever reason the launch script (via openstacksdk) was unable to magically determine that it should request a fip, so direct attaching on the provider network was easier for bootstrapping purposes | 20:00 |
| clarkb | I'm happy either way, but not sure how to resolve it either direction from where we currently are? | 20:00 |
| donnyd | I will detach the private v4 network, attach a new interface that is directly connected to wan. I just need you to make sure that the two interfaces come up | 20:01 |
| clarkb | si it not possible to do it with one interface? | 20:01 |
| clarkb | (two will work but we may want to rebuild in that case) | 20:02 |
| donnyd | Well that would seem to be the issue with this setup | 20:03 |
| *** jcoufal has quit IRC | 20:04 | |
| clarkb | fwiw the floating IPs did totally work fine the first time around | 20:04 |
| donnyd | Before I was just lying to neutron and essentially redirecting traffic on the edge. well this is the proper way to do ipv6, but not exactly perfect for dual stack | 20:04 |
| clarkb | donnyd: the proper way is with two interfaces? vexxhost instances have a single itnerface that is dual stack | 20:04 |
| clarkb | (I mean both ways work not sure one is better than the other) | 20:04 |
| *** diablo_rojo has quit IRC | 20:06 | |
| clarkb | donnyd: fungi: so floating IP attachemnts were not working on launch when tried previously. We then updated to use direct attached public IPs, but then the cloud was updated to make that not work anymore? So the correct way is to use a floating IP again? | 20:06 |
| clarkb | if I'm caught up and that is accurate should we just launch node a new mirror and make sure the floating ip and all of that is configured proeprly for us by the cloud and ansible/sdk | 20:07 |
| clarkb | mordred: ^ you might want to weigh in too | 20:07 |
| fungi | for the record, here's what the interface and routing looks like on a vexxhost mirror http://paste.openstack.org/show/755136/ | 20:07 |
| fungi | fun that we have a docker0 interface on that mirror | 20:09 |
| fungi | must be left over from the great dockering | 20:09 |
| donnyd | well fips should work fine now | 20:13 |
| donnyd | its was a permissions thing before | 20:14 |
| *** hamzy has quit IRC | 20:14 | |
| clarkb | cool in that case my vote is lets just build a new one and make sure that things are configured properly from start to finish and just use FIPs | 20:14 |
| donnyd | ok | 20:15 |
| *** ramishra has quit IRC | 20:15 | |
| donnyd | works for me | 20:15 |
| *** xek has quit IRC | 20:15 | |
| *** e0ne has joined #openstack-infra | 20:16 | |
| clarkb | cool I'm going to figureo ut lunch but then I can work on that | 20:16 |
| donnyd | When you launch the mirror, do not boot from volume. That instance will hit a dedicated node, so it won't pick up it's dedicated storage if you boot from volume | 20:18 |
| donnyd | enjoy your foods | 20:18 |
| fungi | gonna go run a quick grocery errand and be back as soon as i can | 20:18 |
| clarkb | donnyd: noted | 20:18 |
| clarkb | donnyd: do I need to call it something specific to get the special dedicated node? or are all instances from that project/user sent to the one hypervisor? | 20:19 |
| donnyd | no, that flavor lands on a node that has the proper resources for it | 20:19 |
| donnyd | And I also disable all the other hypervisors for good measure | 20:19 |
| donnyd | If this cloud did something else, i wouldn't worry about it.. but it doesn't, so i want to see how fast we can make it go... and if any of this makes a measurable impact | 20:20 |
| *** dpawlik has joined #openstack-infra | 20:25 | |
| clarkb | gotcha | 20:25 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: js: correct action names https://review.opendev.org/673680 | 20:28 |
| *** dpawlik has quit IRC | 20:35 | |
| *** pabelanger has quit IRC | 20:37 | |
| *** mhu has quit IRC | 20:38 | |
| *** sgw has joined #openstack-infra | 20:39 | |
| *** e0ne has quit IRC | 20:41 | |
| *** bobh has joined #openstack-infra | 20:42 | |
| *** e0ne has joined #openstack-infra | 20:42 | |
| *** tdasilva_ has quit IRC | 20:46 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: use opendev image building system for zuul-operator test https://review.opendev.org/673020 | 20:46 |
| *** eharney has joined #openstack-infra | 20:46 | |
| *** tdasilva_ has joined #openstack-infra | 20:46 | |
| ianw | clarkb: i just realised i logged the status into the wrong channel last night, but i had a poke at elasticsearch. there were hung puppet processes taking up 100% cpu (since 2018) | 20:47 |
| *** hamzy has joined #openstack-infra | 20:47 | |
| ianw | and then a whole bunch of shards were stuck in INITALIZING state and had been for days. so i shutdown and rebooted the nodes one by one, to try and redistribute them, but it didn't work | 20:48 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-jobs master: Add auth config to kubelet user for buildset registries https://review.opendev.org/673351 | 20:48 |
| *** bobh has quit IRC | 20:48 | |
| ianw | so i ended up setting the replications to 0, then setting it back | 20:49 |
| ianw | anyway, the cluster is now green status (was yellow with all the unallocated shards) and i feel like log searching is much more performant | 20:49 |
| *** igordc has quit IRC | 20:50 | |
| clarkb | ianw oh interesting | 20:52 |
| clarkb | ianw did you set the replicasback? | 20:52 |
| ianw | clarkb: yep to 2 | 20:52 |
| clarkb | I think we may inly need 1 replica? | 20:52 |
| clarkb | or wasit 1-> 2? | 20:53 |
| ianw | yeah, it had two replicas ... which i didn't think wasn't strictly necessary, but i wondered if logstash was doing that or something | 20:53 |
| clarkb | basically we should have two copies of the data not 3 | 20:53 |
| clarkb | it allows us to do rolling restarts | 20:53 |
| clarkb | but other than that isnt super useful for our data | 20:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: js: don't transform json logfiles https://review.opendev.org/673688 | 20:54 |
| ianw | clarkb: actually no sorry, i set it back to 1, but i think it was at 2 | 20:54 |
| ianw | clarkb: http://paste.openstack.org/show/755139/ | 20:55 |
| clarkb | that paste lgtm. I woudl expect two copies (one replica) | 20:56 |
| clarkb | new fn mirror is launching now | 21:00 |
| ianw | clarkb: yeah, before i started it was like -> http://paste.openstack.org/show/755140/ | 21:00 |
| clarkb | openstack.exceptions.SDKException: Error in creating the server (no further information available) | 21:00 |
| clarkb | huh so the replicas weren't getting made weird | 21:01 |
| clarkb | donnyd: ^ that exception mean anything to you? maybe I need to delete the old mirror to have enough space for the new one? | 21:01 |
| ianw | clarkb: i dunno, there were messages in some about getting locks and stuff, but then timeouts ... googling it was a hot mess of non answers | 21:02 |
| ianw | or, "we fixed that, please upgrade" :) | 21:02 |
| clarkb | ya {'message': 'No valid host was found. ', 'code': 500, 'created': '2019-07-30T21:00:20Z'} says server show so I bet I have to delete the existing mirror | 21:02 |
| clarkb | ianw: ya doing what you did was likely the best thing | 21:02 |
| clarkb | basically tell it to try again | 21:02 |
| clarkb | fungi: donnyd is fn serving jobs now? if so I'll need to disable it before I delete the mirror and try again | 21:03 |
| ianw | clarkb: anyway, that only started because i was pulling stats to see the osc timing :) | 21:03 |
| clarkb | ianw: oh neat. I was wondering if anyone would look at that more closely | 21:03 |
| donnyd | it is not | 21:04 |
| clarkb | ok I'm going to delete the current mirror then, then launch a new one | 21:05 |
| *** bobh has joined #openstack-infra | 21:05 | |
| donnyd | have to delete the old one first | 21:05 |
| *** bobh has quit IRC | 21:05 | |
| clarkb | ya deleted both servers (the active and the ERROR'd nodes) | 21:06 |
| clarkb | donnyd: now: openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://openstack.fortnebula.com:13696/v2.0/floatingips.json, Bad floatingip request: Network 3a137b0c-f5da-4ed8-b44d-912a2557aed7 does not contain any IPv4 subnet. | 21:07 |
| donnyd | one sec | 21:07 |
| donnyd | Im a non admin in that project and i can see the subnet | 21:09 |
| donnyd | https://www.irccloud.com/pastebin/Im6bZb0P/ | 21:09 |
| clarkb | that is a different network uuid | 21:10 |
| clarkb | let me show some networking info from our side | 21:10 |
| donnyd | those are the subnets | 21:10 |
| clarkb | donnyd: the third column | 21:10 |
| donnyd | https://www.irccloud.com/pastebin/CHDSbDs0/ | 21:10 |
| donnyd | Oh, yea.. .you want to use the openstackci network | 21:10 |
| clarkb | gotcha ok | 21:10 |
| clarkb | now running with --network openstackci | 21:12 |
| donnyd | and i think the fip we want is already resevered, so i may need to drop it if the automation doesn't grab it up | 21:12 |
| donnyd | its allocated to the project is what i mean | 21:12 |
| clarkb | donnyd: we can also just update the dns records (I have to use a different server name anyway) | 21:12 |
| clarkb | openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://openstack.fortnebula.com:13696/v2.0/floatingips.json, Bad floatingip request: Network 3a137b0c-f5da-4ed8-b44d-912a2557aed7 does not contain any IPv4 subnet. I still get that | 21:13 |
| clarkb | mordred: ^ this seems like an sdk bug | 21:13 |
| clarkb | if I've told it to use a network that isn't that network why is it trying to use that network? | 21:13 |
| clarkb | If I list/show the server I see that it is on the openstackci network | 21:14 |
| clarkb | so why does it try to use public_v6 with the fip? | 21:14 |
| donnyd | maybe because it sees that as external as well | 21:15 |
| donnyd | let me try removing the fip allocation so the automation will try itself | 21:15 |
| clarkb | ok | 21:15 |
| donnyd | hrm... look like the fip was still associated with the old instance | 21:16 |
| donnyd | ok, its done. try #3 | 21:16 |
| donnyd | :) | 21:16 |
| clarkb | rerunning now | 21:16 |
| clarkb | after I delete the server | 21:16 |
| *** e0ne has quit IRC | 21:16 | |
| clarkb | donnyd: same error | 21:18 |
| clarkb | and I don't see an FIPs under floating ip list so I think it is trying to create an fip on that network despite me saying use the other network | 21:18 |
| donnyd | well there is an external v4 and external v6 network | 21:18 |
| donnyd | so it must be favoring the v6 network | 21:18 |
| clarkb | earlier monty had said it just picks the first network it sees if you don't specify a network | 21:19 |
| clarkb | and public_v6 is listed first for me, but I am also specifying a network | 21:19 |
| clarkb | let me try using the network uuid instead of the name | 21:19 |
| donnyd | ok | 21:21 |
| clarkb | same thing. Ok now i have to read code and figure out why it is ignoring my specific network request | 21:21 |
| *** pkopec has quit IRC | 21:23 | |
| *** gyee has quit IRC | 21:23 | |
| clarkb | _add_auto_ip() doesn't accept network information at all | 21:23 |
| clarkb | mordred: ^ what is the correct way to deal with this then? | 21:23 |
| clarkb | create_floating_ip() which _add_auto_ip() calls does accept a network parameter | 21:24 |
| clarkb | ok its defniitely doing the find first network pattern | 21:28 |
| donnyd | I can fix it on my end | 21:30 |
| donnyd | i can just make the v6 network look shared instead of external | 21:30 |
| clarkb | donnyd: or hide it from our tenant? | 21:31 |
| *** sreejithp has quit IRC | 21:31 | |
| donnyd | yea, i can do that | 21:31 |
| clarkb | openstackci-fortnebula can see public_v6 and public_v4 but openstackzuul-fortnebula only see public_v6 | 21:31 |
| clarkb | I think for both maybe we want to set it up so they only see openstackci network? | 21:31 |
| clarkb | (or do the test nodes need to use a different network?) | 21:31 |
| donnyd | well there has to be the public_v4 network, that is where FIP's come from | 21:32 |
| clarkb | ah | 21:32 |
| donnyd | but, the public v6 network does not need to show up as external in that tenant | 21:32 |
| clarkb | I also think that sdk should pass in the network it was told to use in this case | 21:33 |
| clarkb | oh I get it now though | 21:33 |
| clarkb | its looking for the network to allocate ipv4 fips out of so it wants to find public_v4 | 21:33 |
| clarkb | the real bug is sdk should filter out any networks without an ipv4 subnet | 21:34 |
| donnyd | right, we want to to see only one external network | 21:34 |
| donnyd | or just connect it after the fact | 21:34 |
| clarkb | well in theory it is all suppsoed to work in one go | 21:34 |
| clarkb | mordred: ^ I know that will require sdk to look up subnet info, but may be worthwhile to make this work better | 21:35 |
| ianw | oh, looks like iad opendev.org mirror just paniced | 21:35 |
| corvus | Download error on http://mirror.iad.rax.opendev.org/pypi/simple/pbr/: timed out -- Some packages may not be found! | 21:35 |
| corvus | yeah i just got that in a job | 21:36 |
| clarkb | donnyd: let me know when you think public_v6 is hidden properly | 21:36 |
| clarkb | and I'll give it another go | 21:36 |
| donnyd | ok | 21:36 |
| corvus | ianw: you handling iad? | 21:37 |
| donnyd | clarkb: so the v6 network will be made available after (i think we already did this one this way once | 21:40 |
| donnyd | So it will get a v6 ip address, it just won't be routable until after provisioning | 21:40 |
| ianw | corvus: yep, back up now | 21:40 |
| donnyd | ok, should be good to go now | 21:40 |
| ianw | we can probably take it out of rotation, we have a ton of logs from the last few days now for the offline volume issue | 21:41 |
| ianw | we'll need to really investigate this fscache issue now i think | 21:41 |
| ianw | i'll stop the audit logging on the afs servers, take it out and send some updates in a bit | 21:41 |
| clarkb | donnyd: will it not be ipv6 via the 16937b20-df93-44e0-bbd8-df452d52111d subnet? this is the openstackci_v6_subnet | 21:42 |
| clarkb | donnyd: I'm giving launch a try | 21:42 |
| clarkb | donnyd: fungi mordred fyi https://storyboard.openstack.org/#!/story/2006310 | 21:42 |
| donnyd | yea, it will get a v6 ip | 21:42 |
| clarkb | that is the sdk bug | 21:42 |
| *** gyee has joined #openstack-infra | 21:42 | |
| donnyd | but that IP address won't route traffic until provisioning is finished | 21:43 |
| donnyd | well i mean won't have a public route | 21:43 |
| clarkb | I see | 21:43 |
| donnyd | so we won't have to change anything in the router, but bgp won't advertise the route for the v6 network that the mirror sits on until after provisioning has cleared | 21:44 |
| clarkb | seems to be working now. it is trying to ssh to that ipv4 address | 21:45 |
| fungi | okay, back now, catching up | 21:45 |
| clarkb | hopefully the next time we have to do this the sdk bug above will be fixed and it will just work (tm) | 21:45 |
| clarkb | oh bah it failed to build because ipv6 pings failed. But we have a switch to turn that off so I'll rerun it and it should actually work this time | 21:46 |
| donnyd | LMK what else i need to do | 21:47 |
| clarkb | donnyd: I think you are good, we have checks in our launch scripts that ping ipv6 addrs because we've had trouble where ipv6 never works for some hosts in some clouds | 21:48 |
| clarkb | in this case we expect it to not work so I've just passed the flag that says don't check ipv6 even though you have an addr | 21:48 |
| clarkb | and it should work this time | 21:48 |
| clarkb | ansible is running the base playbook now | 21:49 |
| donnyd | Well from the work that was happening earlier, all the signs looked good on my end.. so i am hoping we get to move on from this today to bigger and better things | 21:49 |
| donnyd | the new control plane is working much better than the old one | 21:49 |
| clarkb | fatal: [mirror01.regionone.fortnebula.opendev.org]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"2001:470:e045:2:f816:3eff:feac:7c78\". Make sure this host can be reached over ssh", "unreachable": true} | 21:50 |
| clarkb | I thinkt here is a bug in our flag to ignore ipv6 | 21:50 |
| donnyd | ok, i can make v6 work | 21:51 |
| clarkb | no I see the bug | 21:51 |
| clarkb | I'll fix it, I think we are still good | 21:51 |
| donnyd | i have about 6 more ways to shave the v6 yak | 21:52 |
| *** slaweq has quit IRC | 21:53 | |
| ianw | fungi: speaking of which, as i think one of the the ipv6 experts, i'd love your thoughts if a person has a /56 from their isp, but wants a VM on their laptop to have ipv6 capabilities BUT also handle being plugged and unplugged from the docking station from wired to wifi transparently | 21:54 |
| ianw | with ipv4 and nat it just works. with ipv6 it just doesn't work :/ | 21:55 |
| donnyd | with ipv6 you just don't nat | 21:56 |
| donnyd | you route | 21:56 |
| donnyd | what part of ipv6 doesn't work switching from network to network? | 21:57 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use public_ipv4 addr when ignoring ipv6 https://review.opendev.org/673701 | 21:58 |
| donnyd | i run bgp everywhere i want to not hate myself when working with v6 | 21:58 |
| clarkb | hrm I thought taht was the fix but public_ipv4 doesn't exist? | 21:59 |
| donnyd | public_v4 | 21:59 |
| *** bobh has joined #openstack-infra | 21:59 | |
| *** bobh has quit IRC | 21:59 | |
| clarkb | ah ok zuul calls it public_ipv4 | 21:59 |
| donnyd | oh i thought you were talking about the network name | 22:00 |
| *** diablo_rojo has joined #openstack-infra | 22:00 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use public_v4 addr when ignoring ipv6 https://review.opendev.org/673701 | 22:01 |
| clarkb | donnyd: well sdk docs say it is the name of the var I want too :) | 22:01 |
| donnyd | can't beat that with a stick | 22:03 |
| clarkb | and ansible is working now | 22:03 |
| donnyd | can you ping me when its good to go, and i will hook up the v6 side to bgp and we will be off to the races | 22:04 |
| clarkb | will do | 22:04 |
| donnyd | need to get some foods.. be back later | 22:04 |
| fungi | ianw: sounds like you need to be running a router announcement daemon from your gateway there. i would start by carving that /56 cidr up into /64 networks | 22:04 |
| fungi | and then give your router an internal address in one or more of the /64 networks | 22:05 |
| donnyd | yes, ipv6 traffic is routed | 22:06 |
| donnyd | and advertised | 22:06 |
| donnyd | routing protocols make life easier, but aren't strictly required.. the old control plane for FN had no dynamic routing | 22:07 |
| fungi | depending on the operating system the advertisement daemon could have a variety of names. on my (openbsd) gateways i run rtadvd and tell it what prefix to announce | 22:07 |
| donnyd | oh openbsd will surely support what you are looking to do | 22:08 |
| ianw | fungi / donnyd: so i do that (well i have one /64 doing RA). but i can't add the wifi card to a bridge at all | 22:08 |
| fungi | yeah, openbsd was merely an example. i have no idea what ianw is running on his routers | 22:08 |
| donnyd | so your VM's sit behind something? | 22:09 |
| *** jamesmcarthur has joined #openstack-infra | 22:09 | |
| donnyd | Libvirt? | 22:09 |
| donnyd | or virtualbox? | 22:09 |
| ianw | donnyd: yep, just my "work vm" which i'm using libvirt/qemu/kvm for | 22:09 |
| donnyd | oh, well doesn't libvirt support ipv6 networks? | 22:10 |
| ianw | not that i could figure out in the same way as "pull your laptop out of the docking station and things just keep working" which is where i'd like to be :) | 22:10 |
| donnyd | things should just keep working | 22:11 |
| donnyd | libvirts network in this case would route instead of nat | 22:11 |
| donnyd | and you can surely break a /64 down into multiple networks internally | 22:11 |
| donnyd | I am going to find out | 22:12 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Networking got weird in fn so we rebuilt the mirror https://review.opendev.org/673703 | 22:12 |
| clarkb | fungi: donnyd ^ ok I think it is done and happy. That is the dns update. Working on the ansible inventory update now | 22:12 |
| donnyd | ok cool. Thanks for your efforts today ( and every day) | 22:13 |
| donnyd | ianw: so your use case is you have a single /64 from your provider and you need to do nat like things in an instance on your machine | 22:13 |
| donnyd | and fungi you have a /56, so you can break down publically routable ipv6 networks and hand them to your vm's | 22:14 |
| donnyd | just trying to make sure i know what you are both looking for | 22:14 |
| fungi | oh. i don't have a /56, ianw does | 22:15 |
| donnyd | Do you want the vms to be inbound accessible (globally routable)? | 22:15 |
| ianw | donnyd: i have a /56, but just one /64 configured ATM. i want the vm in my laptop to have a global ipv6 address, and work transparently if it's plugged into the docking station via wired ethernet, and via wifi when not | 22:15 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Replace the fn mirror again https://review.opendev.org/673704 | 22:15 |
| *** panda has quit IRC | 22:15 | |
| clarkb | fungi: donnyd ^ and that should update the inventory and have ansible configure the host | 22:15 |
| donnyd | are your wifi and lan on different interfaces (L@) | 22:15 |
| clarkb | donnyd: I think you are good to do the ipv6 change if you need to | 22:15 |
| donnyd | are your wifi and lan on different interfaces (L2) | 22:15 |
| * fungi has no local ipv6 problems other than having no local broadband providers who know wth they're doing so he has to gre a /48 from a tunnel broker | 22:16 | |
| clarkb | ianw: fungi also https://review.opendev.org/#/c/673701/ was necessary to launch node that host | 22:16 |
| donnyd | fungi: HE? | 22:16 |
| clarkb | care to review that one too? | 22:16 |
| fungi | donnyd: yep | 22:16 |
| donnyd | Works great for me... I have a ridiculously expensive business connection.. and i gets none of the v6 love from my provider either | 22:17 |
| ianw | donnyd: yep, the docking station has the wired ethernet and the wifi is, well, the wifi card. | 22:17 |
| clarkb | my home isp just got bought by another isp | 22:18 |
| clarkb | I'm hoping that if they make any improvements native ipv6 is one of them | 22:18 |
| *** panda has joined #openstack-infra | 22:18 | |
| donnyd | ianw: And they both connect to a switch I assume? your WAP has to be connected somewhere. Is it on the same L2 as your hard wires | 22:18 |
| ianw | donnyd: yep, just a standard setup like that | 22:19 |
| *** diablo_rojo has quit IRC | 22:19 | |
| ianw | in .au we now have the "national broadband network" where NBN co provides the wires, and you choose a retail ISP ... I chose the one that provides ipv6 | 22:20 |
| donnyd | So in the network world there should be no difference between interfaces.. You can just set a static ipv6 address on your virb0 and then point routes to it | 22:20 |
| ianw | but i can't add the wifi card to a bridge | 22:20 |
| donnyd | https://usercontent.irccloud-cdn.com/file/6n0aep36/Screenshot%20from%202019-07-30%2018-21-55.png | 22:22 |
| donnyd | works from here, so I can assume it's working everywhere | 22:22 |
| clarkb | donnyd: my test works too | 22:23 |
| donnyd | so we just need to wait for DNS to populate and i can turn the nodepool back up to 11 | 22:23 |
| clarkb | infra-root https://review.opendev.org/#/q/topic:fn-mirror-replacement reviews very much appreciated to get fn back in business | 22:24 |
| clarkb | donnyd: we need dns updated then ansible to run with the inventory updates (that is what will configure afs, apache, lets encrypt and so on) | 22:24 |
| clarkb | the chagnes are in that link I just pasted | 22:24 |
| donnyd | I C | 22:25 |
| donnyd | I see what you are saying ianw | 22:25 |
| *** ekultails has quit IRC | 22:25 | |
| *** eharney has quit IRC | 22:26 | |
| donnyd | on v4 nat happens at virb0 to whatever and in v6 the info would need to be know in advance.. so in other words your edge router needs to know how to get to virb0 whether you are on lan or wifi.... I wonder if this is something keepalived could handle | 22:27 |
| donnyd | keepalived should be able to handle your static ipv6 address, and just prefer to be on lan when connected... then you can advertise a single route from your edge to that keepalived maintained ip | 22:28 |
| donnyd | i will have to do some tinkering and get back to you | 22:28 |
| *** yamamoto has joined #openstack-infra | 22:29 | |
| *** yamamoto has quit IRC | 22:33 | |
| donnyd | ianw: laptop distro? | 22:35 |
| *** jamesmcarthur has quit IRC | 22:39 | |
| *** armax has quit IRC | 22:41 | |
| *** mriedem has quit IRC | 22:45 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Attempt to report CORS and related errors https://review.opendev.org/673707 | 22:47 |
| corvus | fungi: based on https://github.com/merklecounty/rget/issues/1 and https://github.com/merklecounty/rget/blob/master/rgetwellknown/wellknown.go#L118 i think i understand the githubbiness of rget | 22:50 |
| corvus | i think the main thing is the conversion of a given file URL to a hostname | 22:51 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Cleanup extra /'s in manifest render view https://review.opendev.org/673605 | 22:51 |
| corvus | it sort of looks like they wanted a canonical string for a file, which of course can't have "/" in it, but also can't have too many "." in it because letsencrypt has a limitation of 10 domain name components | 22:52 |
| corvus | this is how github releases are translated: https://github.com/merklecounty/rget/blob/master/rgetwellknown/wellknown.go#L72 | 22:53 |
| clarkb | corvus: ya they are converting urls to cert names | 22:53 |
| ianw | donnyd: fedora | 22:53 |
| corvus | so that ends up being "some_tag_without_dots.reponame.orgname.github.com" | 22:53 |
| corvus | i think their proposal is that there should be a .well-known/$filename which describes how a site should perform that translation | 22:54 |
| mnaser | are we currently in a state where we can add new projects right now? | 22:54 |
| *** bobh has joined #openstack-infra | 22:54 | |
| * mnaser wanted to add something to the vexxhost namespace on opendev | 22:54 | |
| clarkb | mnaser: yes | 22:55 |
| mnaser | sweet, thanks | 22:55 |
| clarkb | mnaser: all backends are replaced now so we are clear to add projects again | 22:55 |
| corvus | so i guess tarballs.opendev.org would have ".well-known/rget.txt" which has some content equivalent to those regexes, and maybe outputs something like "some_tag_without_dots.nova.openstack.openstack.org" | 22:55 |
| clarkb | corvus: hrm do they enforce dns name limitations too? | 22:56 |
| corvus | that irks me a bit; it seems that it would be much nicer to not require a file like that, and instead be able to translate any filename into a certname | 22:56 |
| corvus | clarkb: whatcha mean? | 22:56 |
| corvus | clarkb: oh, sorry, my examples all should have used '-' not '_' | 22:56 |
| clarkb | corvus: 63 characters max between dots and 255 total in the full name | 22:56 |
| *** tkajinam has joined #openstack-infra | 22:57 | |
| clarkb | but ya also character limitations too | 22:57 |
| clarkb | I guess this is the downside to not using a blockchain | 22:57 |
| corvus | clarkb: i bet so since there are certs involved | 22:57 |
| clarkb | have to operate within the parameters of the certificate transparency report | 22:57 |
| fungi | it's not entirely clear to me why they can't sign some data that hashes the url and checksum together instead, but i'm no expert in the field | 22:57 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Networking got weird in fn so we rebuilt the mirror https://review.opendev.org/673703 | 22:57 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: projects: add vexxhost/openstack-monitoring https://review.opendev.org/673708 | 22:57 |
| mnaser | ^ if possible :) | 22:57 |
| clarkb | fungi: because they are tracking it via the certificate transparency project | 22:58 |
| clarkb | fungi: so you have to have unique identifiers that can go into there | 22:58 |
| fungi | and yeah, i guess it all boils down to needing to turn it all into a "domain" so as to fit in an x.509 cn field | 22:58 |
| clarkb | yup | 22:58 |
| *** eernst has quit IRC | 22:59 | |
| corvus | so why can't you just take the path portion of the url and convert the entire thing into a single field? so https://tarballs.openstack.org/nova/nova-123.tgz becomes nova-nova-123-tgz.tarballs.openstack.org ? | 23:00 |
| corvus | is the concern that multiple paths might produce the same output there? | 23:00 |
| clarkb | corvus: you'd have to split it with a . every 63 characters, but I think that should be ok? | 23:00 |
| corvus | nova/nova-123.tgz and also nova-nova-123.tgz | 23:00 |
| clarkb | oh | 23:00 |
| *** rcernin has joined #openstack-infra | 23:01 | |
| *** rfolco|ruck has quit IRC | 23:02 | |
| corvus | but is that so bad? i mean, that's not likely to happen by accident. perhaps something malicious could be done with that? | 23:04 |
| clarkb | I think the worst case is that you have a confusing set of records that you might have to sort through. I guess someone validating one and not the other path would get an invalid checksum result | 23:04 |
| corvus | speaking of that... if the .well-known file is on the same site, then that sort of suggests that someone could update the rules for how to reduce a name at any time.... that could very likely lead to the same set of problems. | 23:05 |
| clarkb | that is a good point | 23:06 |
| *** eernst has joined #openstack-infra | 23:08 | |
| mnaser | https://review.opendev.org/#/c/673708/ would love another +2 if possible :> | 23:10 |
| *** yamamoto has joined #openstack-infra | 23:11 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** eernst has quit IRC | 23:12 | |
| fungi | yeah, i do feel like hosting the translation rules on the same site as the files being referenced reintroduces some of the problem rget is trying to solve | 23:13 |
| *** stevebaker is now known as stevebaker2 | 23:13 | |
| fungi | i initially envisioned losslessly encoding the full url to each artifact, but that's not going to be possible for embedding into a x.509 cert cn | 23:14 |
| fungi | unless the url can itself be turned into a checksum | 23:14 |
| fungi | sha2-256 the url, break it up with . at the right intervals | 23:14 |
| fungi | client hashes the url and looks up the resulting cert? | 23:15 |
| fungi | it's technically not lossless, but it's as good as | 23:15 |
| corvus | ooh that's nice... | 23:15 |
| *** slaweq has quit IRC | 23:15 | |
| *** michael-beaver has quit IRC | 23:15 | |
| fungi | there's probably some subtle reason that won't work which i'm just not clever enough to imagine at this time of night | 23:16 |
| openstackgerrit | Merged opendev/system-config master: Use public_v4 addr when ignoring ipv6 https://review.opendev.org/673701 | 23:19 |
| fungi | could be further shortened by reducing from base16 encoding to base32 (not base64 because dns names are case insensitive) | 23:19 |
| corvus | fungi, clarkb: i was about to write this reply to the github issue: http://paste.openstack.org/raw/755144/ | 23:19 |
| *** lseki has quit IRC | 23:20 | |
| corvus | should i leave it at that (for now), or append a further suggestion of using sha2-256 on the url? or would you like to do that, fungi? | 23:20 |
| fungi | i'm happy to follow up with the full url checksum suggestion | 23:21 |
| fungi | what you have there lgtm | 23:21 |
| corvus | cool, i'll hit the button on that then | 23:21 |
| clarkb | ya that reads fine to me | 23:21 |
| fungi | mnaser: no initial repo content you want to import, just start with only a .gitreview file? | 23:23 |
| mnaser | fungi: yep! | 23:23 |
| fungi | then in it goes! | 23:23 |
| corvus | https://github.com/merklecounty/rget/issues/1 updated | 23:23 |
| corvus | fungi, clarkb: btw, i did try submitting a url, and some git commits showed up in the database as a result: https://github.com/merklecounty/records/commits/master | 23:25 |
| corvus | the most recent 2 commits are for my submission | 23:25 |
| corvus | looking at the source code for "rget server" i'm not actually sure how it made it that far | 23:26 |
| corvus | oh you know what | 23:26 |
| corvus | i'd wager a nickel that they're running a slightly older version without this patch: https://github.com/merklecounty/rget/commit/b25e6525911db033b096b023c743a43a672a8002 | 23:27 |
| openstackgerrit | Merged ttygroup/gertty master: Only search files from the most recent revision https://review.opendev.org/673370 | 23:28 |
| openstackgerrit | Merged ttygroup/gertty master: Correct file search implementation https://review.opendev.org/673371 | 23:28 |
| donnyd | So we are just waiting for the next ansible run and FN is back in business | 23:29 |
| openstackgerrit | Merged opendev/system-config master: Replace the fn mirror again https://review.opendev.org/673704 | 23:32 |
| *** n-saito has joined #openstack-infra | 23:34 | |
| openstackgerrit | Merged openstack/project-config master: projects: add vexxhost/openstack-monitoring https://review.opendev.org/673708 | 23:37 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Revert "Enable IAD opendev.org mirror" https://review.opendev.org/673710 | 23:38 |
| clarkb | ianw: ^ I approved that as it seems like the sooner that is in the better, but wanted to let you know in case you were only staging that | 23:39 |
| ianw | clarkb: nup, that's good. what i'll do now is maybe write some scripts to do long recursive parallel wgets and see if we can trigger this oops in a more repeatable fashion | 23:40 |
| ianw | it seems unlikely we'll get anywhere until we have something fairly reproducible | 23:41 |
| clarkb | I have accepted mirror01.regionone.fortnebula.opendev.org's host key on bridge | 23:41 |
| clarkb | ansible should run against that and get us an afs in the near future | 23:41 |
| *** betherly has quit IRC | 23:41 | |
| clarkb | ianw: ^ maybe you want to watch that and see if it needs a reboot due to unmounted openafs volumes? | 23:41 |
| clarkb | I'm about to go pay attention to my television for a bit | 23:41 |
| fungi | it gets lonely sometimes | 23:41 |
| ianw | clarkb: ok, will check in on it | 23:41 |
| clarkb | fungi: civic duty | 23:42 |
| ianw | would there be interest in splitting the run_all logs up now that they're separate playbooks into more managable log files? | 23:42 |
| ianw | one for each playbook? easier to check on things like this, but you do loose a bit of global view | 23:42 |
| clarkb | ianw: personally I think I would prefer it if we can clean up the logs so that we reduce the warnings and such | 23:42 |
| clarkb | then if that results in still unreadable logs maybe consider splitting at that point? | 23:43 |
| ianw | yeah there's a lot of ***'s | 23:43 |
| ianw | which i don't think are all necessary. i feel like we had a patch for that, in the ansible timing plugin thing maybe | 23:43 |
| fungi | clarkb: yup, i know that of which ye speak | 23:44 |
| *** armax has joined #openstack-infra | 23:44 | |
| ianw | https://github.com/ansible/ansible/pull/45265 ... i should maybe get back to that, although last time it went into the "changes required" black hole | 23:45 |
| *** dchen has joined #openstack-infra | 23:56 | |
| *** rh-jelabarre has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
