Monday, 2020-10-05

*** DSpider has quit IRC00:12
*** roman_g has joined #opendev01:19
*** ysandeep|holiday is now known as ysandeep02:12
*** redrobot has joined #opendev02:54
*** elod has quit IRC03:35
*** elod has joined #opendev03:37
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Refactor fetch-sphinx-tarball to be executor safe
openstackgerritIan Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround
*** roman_g has quit IRC04:29
*** ykarel has joined #opendev04:33
*** marios has joined #opendev05:00
AJaegermnaser: vexxhost CI is producing POST_FAILURES on zuul-jobs repo like
AJaegerinfra-root, looks dead, I cannot reach it in the browswer06:09
ianwAJaeger: looking06:11
AJaegerthanks, ianw06:12
*** tkajinam has quit IRC06:13
ianwi can get a console link, but can't get up a console06:13
*** tkajinam has joined #opendev06:13
ianwerror 100606:14
ianwOS-EXT-STS:vm_state                  | error06:15
*** ralonsoh has joined #opendev06:16
ianw#status log rebooted which had gone into error state06:17
openstackstatusianw: finished logging06:17
AJaegerianw: it's up again, thanks!06:19
*** eolivare has joined #opendev06:40
*** roman_g has joined #opendev06:42
openstackgerritMerged zuul/zuul-jobs master: update-json-file: add role to combine values into a .json
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Add Fedora 33 support
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Auto find greatest Fedora cloud image sub-release
*** hashar has joined #opendev06:56
*** slaweq has joined #opendev07:03
roman_gGood morning, everyone.07:05
roman_gThe repository ' focal Release' is not signed.07:05
roman_gThis is the message I get from apt on Ubuntu for focal focal-updates and focal-backports.07:06
roman_gIs there anything what I need to do prior to using those mirrors?07:06
roman_gOr is it a bug in mirroring? I'm not very good at apt mirrors structure.07:07
*** andrewbonney has joined #opendev07:12
*** ykarel_ has joined #opendev07:20
*** ykarel has quit IRC07:23
*** ykarel_ is now known as ykarel07:23
*** rpittau|afk is now known as rpittau07:29
*** fressi has joined #opendev07:37
AJaegerroman_g: this is expected, see also
*** tosky has joined #opendev07:42
roman_gAJaeger, got it. Thanks! "[ trusted=yes ]" would help me, as I understand.07:42
*** ykarel_ has joined #opendev07:48
*** ykarel has quit IRC07:51
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** ykarel_ is now known as ykarel08:02
*** Dmitrii-Sh has joined #opendev08:23
openstackgerritJavier Peña proposed openstack/diskimage-builder master: Ensure yum-utils is installed in epel element
*** dtantsur|afk is now known as dtantsur08:32
*** redrobot has quit IRC08:41
*** redrobot has joined #opendev08:43
*** ykarel_ has joined #opendev09:09
*** DSpider has joined #opendev09:10
*** ykarel has quit IRC09:12
*** ykarel_ has quit IRC09:18
*** ykarel_ has joined #opendev09:19
*** ykarel_ is now known as ykarel09:21
*** ykarel has quit IRC09:42
*** sshnaidm|afk is now known as sshnaidm09:47
*** ykarel has joined #opendev09:54
*** marios has quit IRC10:39
*** DSpider has quit IRC10:40
*** priteau has joined #opendev10:44
*** DSpider has joined #opendev11:24
*** marios has joined #opendev11:31
*** priteau has quit IRC11:34
*** priteau has joined #opendev11:35
openstackgerritsean mooney proposed openstack/diskimage-builder master: [WIP] add apk element
openstackgerritsean mooney proposed openstack/diskimage-builder master: [WIP] add alpine element
openstackgerritsean mooney proposed openstack/diskimage-builder master: [WIP] bootloader-alpine-support
openstackgerritsean mooney proposed openstack/diskimage-builder master: [WIP] openssh-server-alpine-support
openstackgerritsean mooney proposed openstack/diskimage-builder master: [WIP] simple-init-alpine-support
fungiroman_g: by way of explanation, we generate our own package repositories by copying packages from official mirrors, which allows us to generate package indices exactly matching the available packages and so not risk mismatches between mirrored indices and mirrored packages. as a result of not using the official signed indices, our options are to either sign them with an unofficial key or not sign them at12:49
fungiall. the latter helps us reinforce that these repository copies are unofficial and not to be used outside our ci environment (previously when we maintained actual mirrors with official signatures on the indices, random people would set them as their production package mirrors instead of an official mirror)12:49
fungiianw: AJaeger: rackspace also flooded us with roughly 50 e-mail messages from their ticketing system letting us know about a host outage impacting translate01, i suppose that was related12:52
AJaegerfungi: I guess so - thanksk13:01
*** Topner has joined #opendev13:13
*** fressi has quit IRC13:26
*** redrobot has quit IRC13:27
*** redrobot has joined #opendev13:31
*** slaweq has quit IRC13:33
*** slaweq has joined #opendev13:39
openstackgerritPierre Riteau proposed openstack/project-config master: Add Review-Priority flag for cloudkitty
*** tkajinam has quit IRC13:49
*** tkajinam has joined #opendev13:49
*** priteau has quit IRC13:52
roman_gfungi, thank you for the explanation. It now makes sense to me, and it appeared to be a right choice of me using those repos only during CI here, and letting users supply their own repos during software use or rebuilds.13:57
*** priteau has joined #opendev13:58
*** slaweq has quit IRC14:47
*** slaweq has joined #opendev14:49
openstackgerritPierre Riteau proposed opendev/irc-meetings master: Add Rafael Weingärtner to CloudKitty meeting chairs
*** ysandeep is now known as ysandeep|away14:52
*** mlavalle has joined #opendev15:05
*** priteau has quit IRC15:11
fungiinfra-root: reminder, the following production databases are going to experience brief (~5 minute) outages around 03:00-05:00 utc friday: paste, survey, review, grafana15:19
fungireview is probably the one to watch most closely... we might need to restart gerrit to get it to reconnect (though i expect it will be fine other than throwing errors for any attempted interactions when disconnected from its db)15:20
fungiclarkb: ^ might be worth a mention during the meeting too15:20
clarkbI'll add it to the announcements on the agenda when I put that together today15:21
clarkbroman_g: fungi: and the reason for going through all of that trouble is debian and ubuntu mirrors have a tendency to get desynced unless they are top level and receive notifications from the distro directly15:23
clarkbthat leads to job failures because packages aren't available due to the lack of synchronization15:23
roman_gclarkb, is it like a design issue?15:24
clarkbroman_g: yes its a fundamental issue with the package repo design15:24
fungithey serve indices listing the versions and checksums of all packages15:24
fungibut depending on the state the mirror is in when you try to copy from it, the packages listed in the index may not be present15:24
clarkbwe work around it by serving a not quite mirror15:24
fungiyeah, basically unless you are part of their official mirror networks, you can't easily create guaranteed a coherent mirror using their indices15:25
roman_g"depending on the state the mirror" - interesting. I get it. I have had issues related to partially synced ubuntu mirrors previously.15:25
clarkbwe do have https enabled on the mirrors now so any newer distro can use https without adding apt https support (its there by default) and that mitigates much of the risk here15:26
clarkb(aiui reprepro is validating what it gets then ssl helps to ensure you're talkign to reprepro)15:26
fungiyep, though also reprepro is creating indices of the exact packages it copied, rather than copying indices and hoping they match the packages15:27
*** hashar has quit IRC15:47
*** rpittau is now known as rpittau|afk16:06
openstackgerritPierre Riteau proposed opendev/irc-meetings master: Fix pep8 failures
*** ykarel is now known as ykarel|away16:16
*** dtantsur is now known as dtantsur|afk16:21
*** tosky has quit IRC16:22
*** hamalq has joined #opendev16:23
*** marios is now known as marios|out16:30
openstackgerritPierre Riteau proposed opendev/irc-meetings master: Fix pep8 failures
*** ykarel|away has quit IRC16:34
*** marios|out has quit IRC16:48
clarkbfungi: catching up on my change backlog this morning. Any chance you have time for and I can keep any eye on them if they get approved16:49
fungisure, can review while i grab some lucnh16:50
clarkb is related to the first one but probably less urgent (we aren't udpating our mirror at all currently :( )16:50
johnsomHi opendev folks, Could we remove the fedora 28 test image: it is by far stale at this point. We can save you some bit space. grin16:51
johnsomA quick code search shows no one is using it. (I wouldn't have expected anyone to have used it for a long time)16:53
clarkbjohnsom: is there any reason to keep it up? for example if I have an old cloud will building a current image work with it?16:53
clarkbthough I guess it is under test-images/16:53
clarkbideally no one is running that in prod?16:54
johnsomWell, we strongly encourage people to not use those in any deployment16:54
johnsomPlus, we don't really recommend fedora images....16:55
*** hashar has joined #opendev17:03
*** andrewbonney has quit IRC17:08
*** eolivare has quit IRC17:10
clarkbfungi: any reason to not rm ? I'm aklog'd and ready to do it if you have no objects17:20
fungii usually just move stuff like that into my homedir for a while in case someone regrets the removal later17:20
fungino objections though17:21
fungior objects for that matter17:21
* fungi is not especially object-oriented17:21
johnsomWe could always recreate the job that builds it, but that hasn't run for over a year17:22
clarkbjohnsom: I've moved it off the afs filesystem. I'll make a note to rm it from my homdir in a couple weeks17:22
clarkbjohnsom: you'll need to wait for a publishing pulse which I think happens every 5 minutes?17:23
johnsomYeah, no rush. Just housekeeping17:23
openstackgerritMerged opendev/system-config master: Exclude private dirs when mirroring fedora
clarkbinfra-root (and other channel lurkers) it is my understanding that docker container ulimits are based on the ulimits for the dockerd process if not otherwise set. We currently set ulimits on at least the gerrit container which will limit the open files to 8192 as well as set core file size to 017:55
clarkbI think we may have taken the file limit from our old puppetry which was used to increase the file limit17:55
clarkbshould we drop the ulimit for file county entirely since the intent was the increase it but now we are decreasing it?17:55
clarkbfungi: ^ this is in prep for testing the offline reindex fwiw since we set ulimits on the daemon container but not the run commands container. I think we're good since we should allow even more files in the run container which will do the offline reindex17:57
openstackgerritMerged opendev/system-config master: Make gitea description update failures nonfatal
fungiclarkb: in theory we can use ulimit to tell us what the actual limit in the container is both before and after18:09
clarkbya I did that locally at least to confirm ehat I think is the drfault behavior18:11
clarkbwe can double check on review-test too18:11
fungioh, good enough for me, but sure we can check it there too18:11
clarkbalso doing 11am lunches ti sync with kids schedule is weird and difficult :)18:11
fungiwhen you get to be old like me, 11am lunch and 4pm dinner will seem normal ;)18:12
*** ralonsoh has quit IRC18:19
jrosseri have a new type of failure when switching from ansible to ansible-base and relying more on ansible collections
jrosserseems ansible-galaxy now supports installing from git+file:// which might make it possible to carry a set of collections in the CI images18:41
clarkbjrosser: we cache the git repos we host in our images and zuul can act as an intermediary with github18:51
clarkbzuul acts like the cache in those cases18:51
jrosserin this case it's things that would have previously been integral to ansible18:52
clarkbI thought those are still included in the `ansible` package on pypi just not in the git repo called ansile?18:52
jrosserbut ansible-base breaks almost everything out into an external collection which needs to be downloaded/installed18:53
clarkbya ansible and ansible-base are distinct18:54
clarkband ansible is supposed to continue to be a kitchen sink if installed via pypi aiui18:54
jrosseryes thats right - though we've had such bad luck with anything we care about being merged (open PR for periods of years for rabbitmq) that we needed the new collections anyway18:55
clarkbfor git repos in general we've tried to avoid caching them arbitrarily We'd never fit everything onto our images if we did. We're happy to cache what we host since the vast majority of code interaction on the test nodes happens there. And we can hook zuul to external sources as well. I know zuul has talked about doing things to better support collections as well but that would be for the ansible layer zuul19:04
clarkbuses to execute jobs. Not sure how that would interact with nested ansible19:04
jrosserok well i just thought it was worth knowing about19:07
jrosserimho increasingly the bugfixes to useful modules will be in the collections19:07
*** Topner has quit IRC19:14
*** tosky has joined #opendev19:29
*** Topner has joined #opendev20:09
*** roman_g has quit IRC20:09
openstackgerritMerged opendev/irc-meetings master: Fix pep8 failures
openstackgerritMerged opendev/irc-meetings master: Update Airship meeting details
*** mnasiadka has quit IRC20:25
*** jroll has quit IRC20:25
*** jroll has joined #opendev20:27
openstackgerritPierre Riteau proposed opendev/irc-meetings master: Add Rafael Weingärtner to CloudKitty meeting chairs
*** slaweq has quit IRC20:50
*** hashar has quit IRC21:14
ianwjrosser: yeah in the system-config gate we run a ansible devel job21:21
ianwso you can already add a few collections as required projects, see project-config21:22
jrosserosa have 'gone early' on 2.10.1 + collections
fungiin the zuulosphere we're still trying to nail down what collections in job definitions look like21:23
jrossertheres a regular burble of the jobs failing somehow inside the ansible-galaxy command when installing the collections21:24
jrossernot much, but enough i've noticed its a thing21:24
fungithere are times when i get the impression we're the only community who gets particular about maintaining a stable ci system and solving tooling problems rather than just expecting folks to keep retrying until it magically works21:25
fungithe biggest spanners in the works seem to be (lack of) reliability of the internet and clouds21:26
clarkbis galaxy still a front end for github?21:27
clarkblooks like the tarballs are hosted in s321:29
fungiso s/github/amazon/21:29
fungior probably more accurately, amazon's cdn21:31
jrosserif theres interest in avoiding constantly downloading collections i'm happy to test it out21:32
jrosseri can probably do that for community.general already by the look of it21:32
*** roman_g has joined #opendev21:53
clarkb woo that updated finally22:16
*** roman_g has quit IRC22:26
*** DSpider has quit IRC22:35
*** qchris has quit IRC22:41
*** qchris has joined #opendev22:53
*** tosky has quit IRC23:04
*** mlavalle has quit IRC23:09
*** Topner has quit IRC23:33

Generated by 2.17.2 by Marius Gedminas - find it at!