| *** ministry is now known as __ministry | 02:37 | |
| *** dasm is now known as dasm|off | 03:08 | |
| *** ysandeep|out is now known as ysandeep | 03:15 | |
| *** ysandeep is now known as ysandeep|afk | 05:11 | |
| *** ysandeep|afk is now known as ysandeep | 05:57 | |
| *** amoralej|off is now known as amoralej | 07:21 | |
| *** ysandeep is now known as ysandeep|lunch | 08:31 | |
| *** jpena|off is now known as jpena | 08:38 | |
| fzzf[m] | hi, I use nodepool with static provider,zuul can't access static node,nodepool.yaml like this https://paste.opendev.org/show/812366/ , CI run job result show https://paste.opendev.org/show/812365/ ,can't reach 172.0.13.48. How should I modifiy. any help would be appreciated. | 09:34 |
|---|---|---|
| *** ysandeep|lunch is now known as ysandeep | 10:06 | |
| *** bhagyashris__ is now known as bhagyashris | 11:26 | |
| *** dviroel|out is now known as dviroel|ruck | 11:30 | |
| *** rlandy|out is now known as rlandy|ruck | 11:38 | |
| dulek | Hello infra! We recently see an elevated number of seemingly random failures in kuryr-kubernetes gates. Had any infra change been made recently? | 11:39 |
| dulek | To give you an example - last time we've suffered like this it was a change that shrinked default swap size of VMs. | 11:40 |
| *** ykarel is now known as ykarel|away | 12:23 | |
| *** dasm|off is now known as dasm | 12:52 | |
| *** amoralej is now known as amoralej|lunch | 13:09 | |
| *** dviroel is now known as dviroel|ruck | 13:13 | |
| sean-k-mooney | frickler: do you happen to know what model of toaster the arm node pool lable run on :) | 13:35 |
| sean-k-mooney | https://review.opendev.org/q/topic:arm-fixes i started pushing some of my local changes | 13:38 |
| sean-k-mooney | but devstack takes like 55 mins to actuly stack on those hosts | 13:38 |
| sean-k-mooney | i havent looked at the ohter arm job fully yet https://review.opendev.org/c/openstack/devstack/+/708317 | 13:39 |
| sean-k-mooney | perhaps some of the system tuning are really really required to get it to be reasonable | 13:39 |
| sean-k-mooney | but the vms seam very underpowered | 13:39 |
| sean-k-mooney | locally i was able to stack and booth without tweaking the hosting vm much ohter then enableing zswap for compressed swap | 13:43 |
| sean-k-mooney | and that is becasue i only have 8G on the macbook air so i could only really give 6G to the vm | 13:43 |
| sean-k-mooney | running ubuntu | 13:43 |
| sean-k-mooney | i have at least on bug in my patches as its selectign the wrong cpu model so that might speed up the tempest part as might adding the extra local pip mirror | 13:44 |
| sean-k-mooney | for install | 13:44 |
| sean-k-mooney | but 55mins is still a lot of ground to make up | 13:45 |
| fungi | dulek: do the jobs in question use centos-8? we did recently switch the centos-8 nodeset to point to our centos-8-stream label last week after the centos mirror operators removed packages for centos-8 at the beginning of the month | 13:51 |
| fungi | sean-k-mooney: have you compared processor series and core count? | 13:52 |
| sean-k-mooney | hehe no but that should be in the test outpu | 13:52 |
| sean-k-mooney | *job output | 13:52 |
| *** ysandeep is now known as ysandeep|afk | 13:52 | |
| sean-k-mooney | fungi: looking at the previous attpemt there was a lot of tweakign to buffers and cches presumable to over come io bottelnecks | 13:53 |
| sean-k-mooney | devstack thend to be very diskio bound so that coudl be the issue | 13:54 |
| sean-k-mooney | fungi: 8 cpus and 8G ram virt-rhel7.6.0 so this might be using qemu emulation for the vms | 13:56 |
| dulek | fungi: No, we're using Ubuntu, but we do build images based on CentOS and we switched too. That's some idea, thank you. | 13:57 |
| *** amoralej|lunch is now known as amoralej | 13:58 | |
| fungi | sean-k-mooney: have you noticed if the performance is consistent between the linaro and osuosl arm nodes? | 13:58 |
| sean-k-mooney | fungi: i have only done one job run so far but i can keep an eye on that | 13:58 |
| fungi | yeah, we currently have two providers for our arm64 nodes, presumably with at least slightly different hardware and hypervisor | 13:59 |
| fungi | also the linaro environment is due to be replaced with newer hardware, though i haven't seen any estimates as to when yet | 13:59 |
| sean-k-mooney | it ran on ubuntu-focal-arm64-osuosl-regionone-002839500 | 13:59 |
| sean-k-mooney | this is kind of just a side project for me right now since i just wanted to see if i could get thigns runnign on my mac and do work if i needed to trael without bringing my big work laptop | 14:01 |
| sean-k-mooney | but i might see if i can get an arm job working in my spare time anyway | 14:02 |
| sean-k-mooney | right now my version is working but timing out so its reallly just a case fo can this be made resonably enough to use | 14:02 |
| sean-k-mooney | its too bad we dont copy/log a couple of files form /proc and /sys for review after like /proc/cpuinfo and /proc/meminfo | 14:05 |
| sean-k-mooney | or dmesg | 14:05 |
| sean-k-mooney | i mean i can add those but i assuemd they were not added before for a reason | 14:06 |
| fungi | we try to collect relevant stats in files within the zuul_info directory | 14:29 |
| fungi | we can add more like cpu count or whatever there if useful | 14:30 |
| *** ysandeep|afk is now known as ysandeep | 14:55 | |
| sean-k-mooney | fungi: we get some fo that info from ansible and some from nova/libvirt but just doing a cp of /proc/{cpuinfo,meminfo,cmdline} to the zuul_info dir would be nice | 15:21 |
| fungi | sean-k-mooney: we record some networking and filesystem info and stick it in files like https://zuul.opendev.org/t/openstack/build/ce460cf79d864512b7a5ae83805a85a5/log/zuul-info/zuul-info.ubuntu-focal.txt | 15:23 |
| fungi | so could include useful details from cpuinfo et cetera if desired | 15:23 |
| fungi | it's generated by a script run by a role in the base job | 15:23 |
| * fungi finds | 15:23 | |
| sean-k-mooney | oh cool ya append to that would work | 15:28 |
| sean-k-mooney | i dont need this often by the way it sjust nice to have form time to time | 15:28 |
| fungi | it happenshere: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/validate-host | 15:29 |
| fungi | extending that with more info about the node should be fairly trivial | 15:29 |
| sean-k-mooney | ack thanks | 15:30 |
| sean-k-mooney | i wont get much time to play with this till next week | 15:30 |
| fungi | recording entire files off the filesystem is a bit much if it's being done in every single job, but adding known useful details from them to that file doesn't add all that much log data, so it's a good compromise | 15:34 |
| sean-k-mooney | i mean they are pretty small but we coudl get the summary form lscpu | 15:47 |
| sean-k-mooney | meminfo is 1.5K cpu info is 11K cmdline is less then 1K | 15:48 |
| sean-k-mooney | so thsoe 3 would be less teh 15KB raw | 15:49 |
| fungi | yeah, /proc/cmdline we'd likely just cat directly | 15:49 |
| fungi | there are a couple of other things like that already in the role | 15:50 |
| fungi | lscpu is a bit verbose but way shorter that /proc/cpuinfo at least | 15:50 |
| fungi | /proc/meminfo isn't too bad | 15:51 |
| sean-k-mooney | we get some of the info in https://zuul.opendev.org/t/openstack/build/ce460cf79d864512b7a5ae83805a85a5/log/zuul-info/host-info.ubuntu-focal.yaml | 15:51 |
| sean-k-mooney | but not nessisarly all of it | 15:52 |
| sean-k-mooney | anyway its not really pressing | 15:52 |
| sean-k-mooney | i dont curertnly see the cpu model name in teh logs we have but i might have missed it | 15:52 |
| fungi | sure, but could be useful when comparing performance between different runs | 15:52 |
| sean-k-mooney | yep by the way i assume runing with mitigations=OFF on teh kernel would be fround upon.... | 15:53 |
| * sean-k-mooney suspectr that would be one way to speed up the ci | 15:53 | |
| sean-k-mooney | not that its slow in general but if some ot the hardware is old its proably also buggy | 15:54 |
| fungi | i want to say we tested that back when the spectre/meltdown concerns were first being raised, and the performance difference wasn't all that noticeable | 15:55 |
| sean-k-mooney | i noticied it on some of my old hardware but not enough to really care about it | 15:56 |
| fungi | the majority of slowdown in most jobs is still attributable to i/o contention, i think | 15:56 |
| sean-k-mooney | ya same | 15:56 |
| sean-k-mooney | i want to play around wiht using zswap at some point to increase swap performance | 15:56 |
| fungi | though in a lot of cases it's being caused by swap thrash, so could be considered a memory pressure concern | 15:56 |
| fungi | and yeah, that might yield interesting results | 15:57 |
| sean-k-mooney | ya so using zswap would help with that | 15:57 |
| sean-k-mooney | effectivly it increase thw swap read/write bandwith by the compression factor | 15:57 |
| sean-k-mooney | at the overhead of cpu of course to d the compression | 15:57 |
| fungi | (and decompression) | 15:58 |
| sean-k-mooney | yep | 15:58 |
| sean-k-mooney | it seam to be working well with my macbook air vm but i have not realy looked at how much of a difference it is making | 15:58 |
| sean-k-mooney | i jsut have nocticed no negitive sideffect | 15:59 |
| *** ysandeep is now known as ysandeep|out | 16:03 | |
| *** priteau_ is now known as priteau | 16:11 | |
| clarkb | sean-k-mooney: fungi: the linaro cloud is last gen packethost arm. There are discussions to bring it to a more modern system of one variety or another. I think osuosl lists their hardware somehwere on their website too | 16:23 |
| clarkb | the exposure of that info to the VMs is basically nil though | 16:23 |
| clarkb | you get like 4 instructions, a cpu clock speed, and bogomips count | 16:23 |
| clarkb | re mitigations I don't know that arm has any like x86 at least not based on the flags in cpuinfio | 16:24 |
| fungi | oh, right, i wasn't thinking of spectre/meltdown mitigations in the scope of arm nodes, i was talking about the x86 ones | 16:30 |
| *** tkajinam is now known as Guest210 | 16:30 | |
| clarkb | I think we're just still not quite there yet on ARM for general workloads. It works great in my phone and my nintendo switch and in people's mac hardware where things are tuned to use cases and specific needs. But on the server side it seems like availability is super limited for the shiny stuff | 16:32 |
| clarkb | I expect that will change over time | 16:32 |
| *** priteau is now known as priteau_ | 16:48 | |
| *** priteau_ is now known as priteau | 16:48 | |
| fungi | it performs well (and at much lower power consumption/heat generation) for appropriate workloads, but it's worth remembering that people have been tuning software to run well on x86 systems for so long that many workloads aren't really appropriate for arm-based systems yet | 16:57 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 17:09 | |
| *** dviroel|ruck is now known as dviroel|ruck|afk | 17:16 | |
| *** jpena is now known as jpena|off | 17:32 | |
| opendevreview | Merged openstack/project-config master: Give perm to release team to delete branches https://review.opendev.org/c/openstack/project-config/+/828371 | 17:52 |
| *** amoralej is now known as amoralej|off | 18:21 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 18:57 | |
| *** dviroel|ruck|afk is now known as dviroel|ruck | 19:19 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Remove Fedora 34 https://review.opendev.org/c/openstack/project-config/+/816933 | 20:56 |
| *** dviroel|ruck is now known as dviroel|out | 21:49 | |
| *** dasm is now known as dasm|off | 21:49 | |
| opendevreview | Merged openstack/project-config master: Remove Fedora 34 https://review.opendev.org/c/openstack/project-config/+/816933 | 21:51 |
| *** prometheanfire is now known as Guest2 | 23:49 | |
| *** osmanlicilegi is now known as Guest0 | 23:49 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!