| *** ociuhandu has joined #openstack-infra | 00:01 | |
| *** markvoelker has quit IRC | 00:02 | |
| *** ryohayakawa has joined #openstack-infra | 00:04 | |
| *** jamesmcarthur has joined #openstack-infra | 00:17 | |
| *** codecapde has quit IRC | 00:20 | |
| *** jamesmcarthur has quit IRC | 00:26 | |
| *** tetsuro has joined #openstack-infra | 00:27 | |
| *** hamalq has quit IRC | 00:36 | |
| *** rfolco has quit IRC | 00:50 | |
| *** factor has quit IRC | 00:54 | |
| *** factor has joined #openstack-infra | 00:54 | |
| *** jamesmcarthur has joined #openstack-infra | 01:02 | |
| *** jamesmcarthur has quit IRC | 01:04 | |
| *** dtantsur has joined #openstack-infra | 01:12 | |
| *** dtantsur|afk has quit IRC | 01:12 | |
| *** Goneri has quit IRC | 01:15 | |
| *** jamesmcarthur has joined #openstack-infra | 01:17 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| *** jamesmcarthur has joined #openstack-infra | 01:26 | |
| *** jamesmcarthur has quit IRC | 01:39 | |
| *** jamesmcarthur has joined #openstack-infra | 01:47 | |
| *** ricolin has joined #openstack-infra | 02:01 | |
| *** ricolin has quit IRC | 02:01 | |
| *** jamesmcarthur has quit IRC | 02:01 | |
| *** jamesmcarthur has joined #openstack-infra | 02:02 | |
| *** gyee has quit IRC | 02:04 | |
| *** jamesmcarthur has quit IRC | 02:06 | |
| *** tdasilva_ has joined #openstack-infra | 02:24 | |
| *** lamt has quit IRC | 02:24 | |
| *** tdasilva has quit IRC | 02:25 | |
| *** tdasilva_ is now known as tdasilva | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:30 | |
| *** gagehugo has quit IRC | 02:36 | |
| *** coreycb has quit IRC | 02:36 | |
| *** mnasiadka has quit IRC | 02:36 | |
| *** coreycb has joined #openstack-infra | 02:38 | |
| *** mnasiadka has joined #openstack-infra | 02:38 | |
| *** gagehugo has joined #openstack-infra | 02:41 | |
| *** jamesmcarthur has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #openstack-infra | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:50 | |
| *** jamesmcarthur has joined #openstack-infra | 02:56 | |
| *** hongbin has joined #openstack-infra | 03:03 | |
| *** hongbin has quit IRC | 03:13 | |
| *** psachin has joined #openstack-infra | 03:20 | |
| *** tetsuro has quit IRC | 03:28 | |
| *** lbragstad_ is now known as lbragstad | 03:37 | |
| *** rcernin has quit IRC | 03:38 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** rcernin has joined #openstack-infra | 03:46 | |
| *** udesale has joined #openstack-infra | 03:52 | |
| *** ramishra has quit IRC | 03:53 | |
| *** ysandeep|away is now known as ysandeep | 04:01 | |
| *** ykarel|away is now known as ykarel | 04:19 | |
| *** tetsuro has joined #openstack-infra | 04:22 | |
| *** markvoelker has joined #openstack-infra | 04:23 | |
| *** tetsuro has quit IRC | 04:25 | |
| *** tetsuro has joined #openstack-infra | 04:26 | |
| *** markvoelker has quit IRC | 04:28 | |
| *** matt_kosut has joined #openstack-infra | 04:31 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** vishalmanchanda has joined #openstack-infra | 04:41 | |
| *** apetrich has quit IRC | 04:43 | |
| *** jtomasek has joined #openstack-infra | 04:50 | |
| *** tetsuro has quit IRC | 05:02 | |
| *** xek has joined #openstack-infra | 05:02 | |
| *** dmellado has quit IRC | 05:12 | |
| *** ysandeep is now known as ysandeep|brb | 05:41 | |
| *** lmiccini has joined #openstack-infra | 05:43 | |
| *** xek has quit IRC | 05:45 | |
| *** eolivare has joined #openstack-infra | 05:51 | |
| *** marios has joined #openstack-infra | 05:55 | |
| *** ramishra has joined #openstack-infra | 06:00 | |
| *** ysandeep|brb is now known as ysandeep | 06:11 | |
| *** dklyle has quit IRC | 06:13 | |
| *** markvoelker has joined #openstack-infra | 06:24 | |
| *** markvoelker has quit IRC | 06:29 | |
| *** diablo_rojo has quit IRC | 06:49 | |
| *** xek has joined #openstack-infra | 06:54 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** xek has quit IRC | 07:01 | |
| *** ociuhandu has quit IRC | 07:05 | |
| *** rcernin has quit IRC | 07:07 | |
| *** rcernin has joined #openstack-infra | 07:08 | |
| *** jcapitao has joined #openstack-infra | 07:13 | |
| *** xek has joined #openstack-infra | 07:14 | |
| *** bhagyashris|pto is now known as bhagyashris | 07:16 | |
| *** rcernin has quit IRC | 07:23 | |
| *** hashar has joined #openstack-infra | 07:24 | |
| *** ralonsoh has joined #openstack-infra | 07:26 | |
| *** yolanda has joined #openstack-infra | 07:26 | |
| *** amoralej|off is now known as amoralej | 07:28 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 07:28 | |
| *** tosky has joined #openstack-infra | 07:28 | |
| *** bhagyashris is now known as bhagyashris|lunc | 07:29 | |
| *** iurygregory has quit IRC | 07:42 | |
| *** ociuhandu has joined #openstack-infra | 07:45 | |
| *** jpena|off is now known as jpena | 07:50 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/reviewstats master: Stop configuring install_command in tox. https://review.opendev.org/694506 | 07:52 |
|---|---|---|
| *** iurygregory has joined #openstack-infra | 08:01 | |
| sshnaidm|ruck | I have "fatal: The remote end hung up unexpectedly" when preparing a node: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_508/738469/2/check/tripleo-ci-centos-7-containers-multinode-train/5085dab/job-output.txt | 08:02 |
| sshnaidm|ruck | can we maybe add retries for such cases? | 08:02 |
| sshnaidm|ruck | are there any known network issues atm? | 08:03 |
| *** markvoelker has joined #openstack-infra | 08:25 | |
| *** ykarel is now known as ykarel|lunch | 08:25 | |
| frickler | sshnaidm|ruck: you would need to move your preparation steps from run to pre, then zuul can retry if it fails | 08:25 |
| sshnaidm|ruck | frickler, yeah, good idea | 08:27 |
| *** tetsuro has joined #openstack-infra | 08:28 | |
| *** xek has quit IRC | 08:29 | |
| *** tetsuro has quit IRC | 08:29 | |
| *** markvoelker has quit IRC | 08:30 | |
| *** Lucas_Gray has joined #openstack-infra | 08:32 | |
| *** alejandrosg has joined #openstack-infra | 08:37 | |
| *** alejandrosg has left #openstack-infra | 08:37 | |
| *** bhagyashris|lunc is now known as bhagyashris | 08:37 | |
| *** Lucas_Gray has quit IRC | 08:39 | |
| *** Lucas_Gray has joined #openstack-infra | 08:41 | |
| *** derekh has joined #openstack-infra | 08:45 | |
| *** apetrich has joined #openstack-infra | 08:47 | |
| *** tetsuro has joined #openstack-infra | 08:51 | |
| *** slaweq has quit IRC | 08:51 | |
| *** Lucas_Gray has quit IRC | 08:54 | |
| *** tkajinam has quit IRC | 08:55 | |
| *** smarcet has joined #openstack-infra | 08:57 | |
| *** priteau has joined #openstack-infra | 09:08 | |
| *** sshnaidm|ruck has quit IRC | 09:14 | |
| *** yamamoto has quit IRC | 09:17 | |
| *** Lucas_Gray has joined #openstack-infra | 09:19 | |
| *** admcleod has quit IRC | 09:21 | |
| *** yamamoto has joined #openstack-infra | 09:22 | |
| *** sshnaidm has joined #openstack-infra | 09:23 | |
| *** gfidente has joined #openstack-infra | 09:27 | |
| *** tetsuro has quit IRC | 09:33 | |
| *** sshnaidm has quit IRC | 09:42 | |
| *** ykarel|lunch is now known as ykarel | 09:49 | |
| *** slaweq has joined #openstack-infra | 09:52 | |
| *** priteau has quit IRC | 09:54 | |
| *** sshnaidm has joined #openstack-infra | 09:55 | |
| *** xek has joined #openstack-infra | 09:57 | |
| *** markvoelker has joined #openstack-infra | 09:58 | |
| *** amoralej has quit IRC | 10:03 | |
| *** markvoelker has quit IRC | 10:03 | |
| *** xek has quit IRC | 10:06 | |
| *** jpena has quit IRC | 10:07 | |
| *** mugsie has quit IRC | 10:07 | |
| *** mugsie has joined #openstack-infra | 10:10 | |
| sshnaidm | I see a lot of errors with setting iptables rules: | 10:14 |
| sshnaidm | "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/var/lib/zuul/builds/7cdd1b201d0e462680ea7ac71d0777b6/untrusted/project_0/opendev.org/zuul/zuul-jobs/roles/persistent-firewall/tasks/persist/RedHat.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: | 10:14 |
| sshnaidm | Persist ipv4 rules\n ^ here\n" | 10:14 |
| sshnaidm | for example: https://fa41114c73dc4ffe3f14-2bb0e09cfc1bf1e619272dff8ccf0e99.ssl.cf2.rackcdn.com/738557/2/check/tripleo-ci-centos-8-containers-multinode/7cdd1b2/job-output.txt | 10:14 |
| sshnaidm | git.opendev is hanging a lot as well today | 10:22 |
| sshnaidm | infra-root ^ | 10:22 |
| *** xek has joined #openstack-infra | 10:25 | |
| *** ryohayakawa has quit IRC | 10:30 | |
| *** xek has quit IRC | 10:31 | |
| frickler | sshnaidm: there's a "MODULE FAILURE" before that error even, looking whether zuul has some log for that | 10:34 |
| frickler | infra-root: lots of "ModuleNotFoundError: No module named \'gear\'" on ze01, not sure whether that may be related | 10:40 |
| *** marios has quit IRC | 10:40 | |
| sshnaidm | frickler, yeah, primary | MODULE FAILURE - iptables-save failed on primary host | 10:41 |
| AJaeger | frickler: ze01 is the containerized ze. | 10:43 |
| AJaeger | So, if that is only on ze01, can you disable it temporarily so that it won't execute further jobs? | 10:44 |
| frickler | AJaeger: I just did that | 10:44 |
| AJaeger | frickler: I saw corvus, clarkb, mordred fixing ze01 - and hopeing no new failures show up ;( | 10:44 |
| AJaeger | frickler: thanks | 10:44 |
| frickler | #status log stopped zuul-executor on ze01 with "docker-compose down | 10:45 |
| openstackstatus | frickler: finished logging | 10:45 |
| frickler | meh | 10:45 |
| frickler | #status log stopped zuul-executor on ze01 with "docker-compose down" to allow for debugging ModuleNotFoundError | 10:45 |
| openstackstatus | frickler: finished logging | 10:45 |
| *** hashar has quit IRC | 10:47 | |
| frickler | infra-root: I also checked that these errors are only on ze01, not on other executors | 10:47 |
| sshnaidm | infra-root many errors when node is dying, like that: https://pastebin.com/cKZAfAY2 | 10:49 |
| sshnaidm | "msg": "Failed to connect to the host via ssh: ssh: connect to host 2001:470:e126:1:f816:3eff:fe6f:f91d port 22: No route to host", | 10:49 |
| frickler | sshnaidm: I'd have assumed that that would be related to the tripleo ovs c8 issue your colleagues were discussing here for the last couple of days | 10:51 |
| *** bandini has joined #openstack-infra | 10:52 | |
| *** ysandeep is now known as ysandeep|afk | 10:53 | |
| bandini | o/ am getting a lot of "error: RPC failed; result=35, HTTP code = 0 fatal: The remote end hung up unexpectedly" when cloning from opendev.org (https://opendev.org/x/browbeat/ for example). not sure if this is known already? | 10:53 |
| frickler | bandini: yes, this has been mentioned, not sure about the reason for that, though | 10:58 |
| bandini | frickler: ack, thanks | 10:58 |
| *** jcapitao is now known as jcapitao_lunch | 11:03 | |
| *** udesale_ has joined #openstack-infra | 11:04 | |
| *** udesale has quit IRC | 11:06 | |
| sshnaidm | frickler, I see nodes disappearing even on centos7-train jobs, so not sure it's related to c8 ovs issues. For example: tripleo-ci-centos-7-containers-multinode-train: https://pastebin.com/96mftQM4 | 11:13 |
| sshnaidm | sometimes console just stops | 11:13 |
| frickler | sshnaidm: well maybe it isn't only c8, but this does look like the undercloud setup might mangle host networking | 11:15 |
| sshnaidm | frickler, maybe, but we have it also in jobs that don't install anything tripleo specific, like job that only builds containers: tripleo-build-containers-centos-8-ussuri: https://pastebin.com/yfz0HpWm | 11:16 |
| sshnaidm | I think it's worth at least to check executor logs.. | 11:17 |
| *** amoralej has joined #openstack-infra | 11:20 | |
| *** xek has joined #openstack-infra | 11:23 | |
| *** marios has joined #openstack-infra | 11:27 | |
| *** bhagyashris is now known as bhagyashris|brb | 11:32 | |
| *** jpena has joined #openstack-infra | 11:32 | |
| amoralej | hi, packstack-upload-git-mirror job in post pipeline for x/packstack has failed with infra issue | 11:35 |
| amoralej | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_704/be69f207b1cd5cd0eb7b390ae543335b3632cd57/post/packstack-upload-git-mirror/7049165/ | 11:35 |
| amoralej | ubuntu-bionic | ssh: Could not resolve hostname github.com: Temporary failure in name resolution | 11:36 |
| amoralej | could it be re-executed? | 11:36 |
| *** ysandeep|afk is now known as ysandeep | 11:39 | |
| *** jpena is now known as jpena|lunch | 11:46 | |
| *** smarcet has quit IRC | 11:50 | |
| *** markvoelker has joined #openstack-infra | 11:55 | |
| *** rfolco has joined #openstack-infra | 12:02 | |
| *** amoralej is now known as amoralej|lunch | 12:03 | |
| *** mordred has quit IRC | 12:04 | |
| *** zxiiro has quit IRC | 12:06 | |
| *** jamesmcarthur has joined #openstack-infra | 12:07 | |
| *** rlandy has joined #openstack-infra | 12:08 | |
| AJaeger | amoralej|lunch: the next merge will sync again - isn't that enough? | 12:09 |
| *** bhagyashris|brb is now known as bhagyashris | 12:10 | |
| *** dtantsur is now known as dtantsur|brb | 12:13 | |
| *** mordred has joined #openstack-infra | 12:13 | |
| *** jcapitao_lunch is now known as jcapitao | 12:17 | |
| *** smarcet has joined #openstack-infra | 12:18 | |
| *** hashar has joined #openstack-infra | 12:22 | |
| *** dciabrin has quit IRC | 12:23 | |
| *** xek has quit IRC | 12:33 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| *** yamamoto has quit IRC | 12:37 | |
| *** jamesmcarthur has quit IRC | 12:39 | |
| *** priteau has joined #openstack-infra | 12:45 | |
| *** ysandeep is now known as ysandeep|afk | 12:47 | |
| smarcet | fungi: clarkb: morning guys when u have some free time could u review https://review.opendev.org/#/c/717359/ ? thx :) | 12:53 |
| *** xek has joined #openstack-infra | 12:54 | |
| *** rlandy is now known as rlandy|training | 12:55 | |
| *** ysandeep|afk is now known as ysandeep | 12:55 | |
| *** ricolin has joined #openstack-infra | 12:55 | |
| clarkb | bandini: frickler: the last time we had that problem it was a cloud specific routing issue between citynetwork and vexxhost | 12:56 |
| clarkb | bandini: frickler: considering that it may be worth trying to quickly rule out that situation again if possible. Clone via ipv6 and ipv4 separately if possible and see if you get the same result. Check if cloning from another location reproduces the results | 12:57 |
| clarkb | sshnaidm: frickler yes the ssh connection issue is related to the workload we think | 12:58 |
| bandini | clarkb: ack, lemme try (note I don't have ipv6 connectivity) and no another location does not seem to reproduce the problem | 12:58 |
| sshnaidm | clarkb, workload..? | 12:59 |
| clarkb | sshnaidm: something that toci quickstart does is crashing network on the test nodes and then zuul cannot ssh in anymore | 13:00 |
| clarkb | sshnaidm: the workload of the job is causing that failure we think | 13:00 |
| *** ricolin_ has joined #openstack-infra | 13:00 | |
| sshnaidm | clarkb, I have various examples from today, including jobs on centos7 and jobs without toci quickstart running | 13:00 |
| clarkb | sshnaidm: so you think this is a different issue with the same symptom? | 13:00 |
| sshnaidm | clarkb, I've caught a few consoles today: https://bugs.launchpad.net/tripleo/+bug/1885701 | 13:01 |
| openstack | Launchpad bug 1885701 in tripleo "Node failures on upstream infra, hosts are lost" [Critical,Triaged] | 13:01 |
| sshnaidm | clarkb, looks like | 13:01 |
| *** ricolin has quit IRC | 13:03 | |
| *** amoralej|lunch is now known as amoralej | 13:06 | |
| amoralej | AJaeger, yep, good enough, no problem | 13:06 |
| *** xek has quit IRC | 13:11 | |
| *** smarcet has quit IRC | 13:11 | |
| *** yamamoto has joined #openstack-infra | 13:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-git-mirror: use retries to avoid races https://review.opendev.org/738187 | 13:14 |
| *** smarcet has joined #openstack-infra | 13:21 | |
| *** yamamoto has quit IRC | 13:25 | |
| clarkb | sshnaidm: frickler: is it possible those jobs were related to shutting down ze01? Yuo've overridden the attempts to be 1 and both of those jobs were running on ze01 around the time that frickler turned it off? | 13:26 |
| clarkb | I think those two events may be related and now that ze01 is off we can ignore that problem until we context switch to debugging that issue | 13:27 |
| sshnaidm | clarkb, I see 2 of 3 were on ze01, one is on ze12.openstack.org | 13:29 |
| sshnaidm | clarkb, maybe it's related actually, I'll rerun again to see | 13:30 |
| clarkb | sshnaidm: the third one looks like the centos8 issue. It ran on centos8 and failed during the deply task and gave back the cannot connect error. Whereas the other two just drop off which looks like an executor being stopped | 13:30 |
| clarkb | normally its safe for us to stop an executor because we retry up to 2 more times | 13:31 |
| clarkb | but you've overriddent that in this specific instance and so get a slgihtly different behavior | 13:31 |
| sshnaidm | clarkb, yep | 13:31 |
| sshnaidm | clarkb, let me check then again when ze01 is down now.. | 13:31 |
| sshnaidm | maybe it was responsible for a part of retries | 13:32 |
| sshnaidm | clarkb, I found this issue quite a lot as well: https://bugs.launchpad.net/tripleo/+bug/1885697 | 13:32 |
| openstack | Launchpad bug 1885697 in tripleo "Jobs fail in upstream infra when setting iptables rules in host prepare" [Critical,Triaged] | 13:32 |
| sshnaidm | clarkb, maybe we can add "retry" there | 13:32 |
| sshnaidm | it's yet before any tripleo thing is coming to | 13:33 |
| sshnaidm | I think frickler was looking into ^ | 13:33 |
| sshnaidm | do we monitor retry limits somewhere in grafana? | 13:37 |
| clarkb | It should get reported like any other result | 13:37 |
| clarkb | (so yes) | 13:37 |
| *** Lucas_Gray has quit IRC | 13:40 | |
| *** xek has joined #openstack-infra | 13:42 | |
| *** nightmare_unreal has joined #openstack-infra | 13:43 | |
| *** Lucas_Gray has joined #openstack-infra | 13:51 | |
| AJaeger | config-core, a couple of reviews, please: https://review.opendev.org/738513 https://review.opendev.org/738378 https://review.opendev.org/737457 | 13:54 |
| *** Goneri has joined #openstack-infra | 13:56 | |
| *** xek has quit IRC | 14:00 | |
| *** dtantsur|brb is now known as dtantsur | 14:14 | |
| brtknr | Hery folks, any idea why the coverage test is failing: https://zuul.opendev.org/t/openstack/build/84bd94628a69408cbb0e6d0e08d9202b | 14:15 |
| *** ysandeep is now known as ysandeep|away | 14:17 | |
| clarkb | brtknr: https://zuul.opendev.org/t/openstack/build/84bd94628a69408cbb0e6d0e08d9202b/log/job-output.txt#883 | 14:18 |
| fungi | brtknr: looks like "No .testr.conf config file" is the error there, coverage jobs apparently want a testr config | 14:18 |
| brtknr | clarkb: fungi: we recently removed it because someone proposed that stestr.conf was enough | 14:24 |
| brtknr | https://zuul.opendev.org/t/openstack/build/84bd94628a69408cbb0e6d0e08d9202b/log/job-output.txt#888 | 14:24 |
| *** smarcet has quit IRC | 14:24 | |
| brtknr | are you sure it isnt anything to do with^ | 14:24 |
| fungi | i didn't look at the script, but assumed it could be a cascade failure due to the earlier command not returning stdout so an empty string was getting substituted into the expression | 14:26 |
| clarkb | fungi: ya that was my assumption as well | 14:29 |
| AJaeger | brtknr: tools/cover.sh uses testr, it should use stestr | 14:35 |
| clarkb | its worth noting that when I set up coverage with testr it didn't require a special script | 14:37 |
| clarkb | you basically replace python testr with cover testr as the test runner in the testr.conf and then combine results | 14:37 |
| clarkb | nova's tox.ini illustrates this | 14:38 |
| *** smarcet has joined #openstack-infra | 14:39 | |
| *** dklyle has joined #openstack-infra | 14:43 | |
| openstackgerrit | Merged zuul/zuul-jobs master: prepare-workspace: Add Role Variable in README.rst https://review.opendev.org/737352 | 14:45 |
| *** xek has joined #openstack-infra | 14:48 | |
| *** ykarel is now known as ykarel|away | 14:48 | |
| *** smarcet has quit IRC | 14:48 | |
| *** smarcet has joined #openstack-infra | 14:55 | |
| AJaeger | brtknr: the cover test failed for the first time the .testr.conf file was removed, so removing it without any other change was a bad idea ;) | 15:00 |
| *** smarcet has quit IRC | 15:00 | |
| *** zxiiro has joined #openstack-infra | 15:01 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Return upload_results in upload-logs-swift role https://review.opendev.org/733564 | 15:01 |
| *** smarcet has joined #openstack-infra | 15:05 | |
| *** psachin has quit IRC | 15:09 | |
| *** lmiccini has quit IRC | 15:14 | |
| AJaeger | gagehugo: I just added you to security-doc-core now so that you can add anybody else (or remove) from security team for review of security-doc repository. | 15:21 |
| AJaeger | gagehugo: https://review.opendev.org/#/admin/groups/347,members | 15:21 |
| gagehugo | AJaeger: thanks! | 15:22 |
| *** hashar is now known as hasharAway | 15:24 | |
| *** mgoddard has quit IRC | 15:28 | |
| *** vishalmanchanda has quit IRC | 15:30 | |
| *** xek has quit IRC | 15:31 | |
| *** mgoddard has joined #openstack-infra | 15:36 | |
| *** yamamoto has joined #openstack-infra | 15:36 | |
| *** gyee has joined #openstack-infra | 15:38 | |
| *** yamamoto has quit IRC | 15:50 | |
| *** hamalq has joined #openstack-infra | 15:50 | |
| *** sshnaidm has quit IRC | 15:51 | |
| *** hamalq_ has joined #openstack-infra | 15:52 | |
| *** sshnaidm has joined #openstack-infra | 15:53 | |
| *** hamalq has quit IRC | 15:55 | |
| clarkb | fungi: do you have an easy way of finding logs for one of those unreachable jobs on the executor? I wonder if that would offer clues as to why post.yaml seems to cause post-logs.yaml from running at all | 15:56 |
| fungi | sure, just a sec | 15:56 |
| fungi | i guess ze01 is still down so i'll look in 02 | 15:57 |
| clarkb | like maybe there is another error from post-logs.yaml | 15:57 |
| *** hasharAway is now known as hashar | 15:58 | |
| fungi | you can get a list of event ids like this: | 15:59 |
| fungi | grep '\[e: .* result RESULT_UNREACHABLE ' /var/log/zuul/executor-debug.log | sed 's/.*\[e: \([0-9a-f]\+\)\].*/\1/' | 15:59 |
| fungi | so event 7c5651397cf24400b3982e74dfddaab9 is the most recent one i see on ze02 | 15:59 |
| fungi | and this gets you the full log for it: | 16:00 |
| fungi | grep '\[e: 7c5651397cf24400b3982e74dfddaab9\]' /var/log/zuul/executor-debug.log | view - | 16:00 |
| clarkb | we now need to find one that has reported its build to zuul | 16:01 |
| clarkb | and has no logs | 16:01 |
| clarkb | might be easier to go look for tripleo jobs that have retry limits posted /me will look from that end | 16:02 |
| fungi | yeah, you can find those in the scheduler log | 16:03 |
| corvus | i just looked for the last 10 reported and don't see any missing logs | 16:03 |
| fungi | but then you'll need to work out from the scheduler log which executor to look on | 16:03 |
| *** jpena is now known as jpena|off | 16:03 | |
| *** marios has quit IRC | 16:04 | |
| clarkb | ya I think that maybe there was some misinterpretation of missing logs between what tripleo is seeing (no host logs) and how tristanC interpreted it? no job-output.txt? | 16:04 |
| clarkb | corvus: I'm seeing the same thing | 16:04 |
| clarkb | tristanC: ^ fyi | 16:04 |
| corvus | i just did the next 10 and still can't find one | 16:04 |
| clarkb | the host specific logs are definitely missing but we do expect that and zuul can't do much about that | 16:04 |
| *** yamamoto has joined #openstack-infra | 16:04 | |
| clarkb | sorry I've been context switching between oepndev event, gitea/haproxy debugging, the centos issue and then thinking about it from the zuul side | 16:06 |
| *** jpena|off is now known as jpena | 16:06 | |
| clarkb | I think if what was described was happening then we should fix something (zuul run loop or our base job) but that isn't happening and instead the focus neesd to be on why the hosts are crashing which I've offered to help address in a few ways | 16:06 |
| *** ociuhandu has quit IRC | 16:08 | |
| fungi | looking back at the earlier examples we were discussing to see if that's consistent | 16:08 |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Add support for virtualenv 20.x https://review.opendev.org/719964 | 16:13 |
| *** grantza has joined #openstack-infra | 16:13 | |
| clarkb | the issue is once things get to the db they've completed successfully enough that tehy filter out the other cases | 16:14 |
| clarkb | and as I just mentioend in #zuul the issue is we're failing in the run phase which short circuits and doesn't run any post | 16:14 |
| clarkb | it will only run post if the failure is due to normal playbook completion | 16:15 |
| fungi | sorry, had an hvac contractor show up with no warning right in the middle of trying to look this up | 16:17 |
| *** sshnaidm is now known as sshnaidm|ruck | 16:18 | |
| fungi | seems wednesday around 12:36z is when we started discussing this in here | 16:19 |
| fungi | the initial failure examples provided by rlandy were https://zuul.opendev.org/t/openstack/build/746066b4c7654e01adc5a9aa532a0da3 and https://zuul.opendev.org/t/openstack/build/6fd612696eab452d990a8b7ea3765655 | 16:22 |
| tristanC | sshnaidm|ruck: clarkb: fwiw we are adding opendev's centos-8-0000070210.qcow2 to a rdo's cloud-image based label so that we can try to reproduce locally | 16:26 |
| *** ociuhandu has joined #openstack-infra | 16:26 | |
| clarkb | tristanC: ok | 16:27 |
| *** ociuhandu has quit IRC | 16:32 | |
| *** amoralej is now known as amoralej|off | 16:34 | |
| *** ociuhandu has joined #openstack-infra | 16:36 | |
| *** udesale_ has quit IRC | 16:39 | |
| *** jcapitao has quit IRC | 16:39 | |
| *** smarcet has quit IRC | 16:40 | |
| *** dychen has joined #openstack-infra | 16:42 | |
| *** smarcet has joined #openstack-infra | 16:44 | |
| *** dchen has quit IRC | 16:45 | |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| *** eharney has quit IRC | 16:50 | |
| *** jpena is now known as jpena|off | 16:52 | |
| *** Limech has joined #openstack-infra | 16:57 | |
| *** derekh has quit IRC | 17:04 | |
| *** eharney has joined #openstack-infra | 17:08 | |
| *** Lucas_Gray has quit IRC | 17:29 | |
| *** yamamoto has quit IRC | 17:31 | |
| *** hashar is now known as hasharAway | 17:40 | |
| *** nightmare_unreal has quit IRC | 17:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use a temporary registry with buildx https://review.opendev.org/738517 | 17:51 |
| *** ricolin_ has quit IRC | 17:53 | |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Add support for virtualenv 20.x https://review.opendev.org/719964 | 17:53 |
| *** TViernion has quit IRC | 17:58 | |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Add support for virtualenv 20.x https://review.opendev.org/719964 | 18:04 |
| *** TViernion has joined #openstack-infra | 18:07 | |
| *** eolivare has quit IRC | 18:09 | |
| *** TViernion has quit IRC | 18:21 | |
| -openstackstatus- NOTICE: Due to a flood of connections from random prefixes, we have temporarily blocked all AS4837 (China Unicom) source addresses from access to the Git service at opendev.org while we investigate further options. | 18:22 | |
| *** TViernion has joined #openstack-infra | 18:27 | |
| *** ralonsoh has quit IRC | 18:33 | |
| *** zxiiro has quit IRC | 18:37 | |
| sshnaidm|ruck | seems like opendev.org still has problems with git? pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://opendev.org/openstack/requirements/raw/branch/stable/ussuri/upper-constraints.txt | 18:45 |
| sshnaidm|ruck | weshay_ruck, ^ | 18:45 |
| clarkb | sshnaidm|ruck: we're discussing it over in #opendev | 18:45 |
| sshnaidm|ruck | clarkb, ack | 18:45 |
| clarkb | long story short it is a ddos of some sort (not sure if malicious or intentional) originating from china unicom IPs (and other chinese ISPs) | 18:46 |
| clarkb | we've blocked the unicom ip ranges and reset the backends and things are stable now | 18:46 |
| clarkb | btu trying to sort out what the next step is considering blocking huge ranges of IPs is less than ideal | 18:46 |
| weshay_ruck | oy | 18:48 |
| *** gfidente has quit IRC | 18:51 | |
| *** hasharAway is now known as hashar | 19:01 | |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 19:02 | |
| *** ianw_pto is now known as ianw | 19:04 | |
| *** yolanda has quit IRC | 19:05 | |
| *** viks____ has quit IRC | 19:22 | |
| *** rlandy|training is now known as rlandy | 19:36 | |
| *** Tengu has quit IRC | 19:39 | |
| *** Tengu has joined #openstack-infra | 19:41 | |
| *** xek has joined #openstack-infra | 19:44 | |
| *** xek has quit IRC | 19:46 | |
| *** sean-k-mooney has joined #openstack-infra | 19:54 | |
| *** hashar is now known as hasharAway | 19:55 | |
| *** dave-mccowan has quit IRC | 19:55 | |
| sean-k-mooney | o/ | 19:56 |
| sean-k-mooney | any idea why im seeing http://paste.openstack.org/show/795408/ | 19:56 |
| sean-k-mooney | i was wondering if it could be related to how the open dev repos are being blocker for some ASNs but form the trace i dont think i should be connecting through one fo the chines isps | 19:57 |
| clarkb | sean-k-mooney: is your source IP hosted by china unicom? | 19:57 |
| sean-k-mooney | no | 19:58 |
| sean-k-mooney | the ipv6 adress is provided by herican electic and my ipv4 adress is form vodafone ireland | 19:58 |
| *** dave-mccowan has joined #openstack-infra | 19:58 | |
| sean-k-mooney | https://whatismyipaddress.com/ip/51.37.16.199 | 19:59 |
| sean-k-mooney | ASN: 15502 | 19:59 |
| clarkb | ya that wasn't in any of the block ranges (I'm also a 51 prefix and I double checked myself because I wanted to make suer I wasn't going to kill ssh) | 19:59 |
| fungi | sean-k-mooney: possible those happened a little earlier when we were restarting git backends... are you still seeing it? | 19:59 |
| sean-k-mooney | im seeing it now yes | 19:59 |
| clarkb | oh wait | 19:59 |
| clarkb | thats to git.openstack.org | 20:00 |
| clarkb | thats the problem | 20:00 |
| sean-k-mooney | yes | 20:00 |
| clarkb | only http and https will work to that name | 20:00 |
| sean-k-mooney | that is what devstack is defaulting too | 20:00 |
| clarkb | we basically had to drop support for git:// when we switched to gitea a while back | 20:00 |
| fungi | oh! now i see the git:// | 20:00 |
| clarkb | since gitea doesn't do that | 20:00 |
| clarkb | and you cano't do redirects for git:// anyway | 20:00 |
| fungi | i thought devstack fixed that like a year ago | 20:00 |
| sean-k-mooney | oh i know whats wrong | 20:01 |
| sean-k-mooney | enable_plugin neutron git://git.openstack.org/openstack/neutron.git | 20:01 |
| *** xek has joined #openstack-infra | 20:01 | |
| sean-k-mooney | this is an old local.conf | 20:01 |
| fungi | that'd do it | 20:01 |
| sean-k-mooney | i copied form one of my backups | 20:01 |
| fungi | yeah, that hasn't worked for a very long time | 20:01 |
| *** xek has quit IRC | 20:02 | |
| sean-k-mooney | yep i could not find my normall sriov local.confs so had to grab an old one form my backups | 20:02 |
| sean-k-mooney | opendev seams to be working fine | 20:02 |
| *** noonedeadpunk has quit IRC | 20:02 | |
| *** noonedeadpunk has joined #openstack-infra | 20:03 | |
| fungi | you're one of the only people to say that today, thanks! ;) | 20:05 |
| *** diablo_rojo has joined #openstack-infra | 20:06 | |
| sean-k-mooney | well io say that and its still trying to clone neutron | 20:06 |
| *** tobiash has quit IRC | 20:06 | |
| sean-k-mooney | so maybe i spoke too soon | 20:06 |
| sean-k-mooney | i might just swap to github for now | 20:07 |
| *** tobiash has joined #openstack-infra | 20:07 | |
| *** yolanda has joined #openstack-infra | 20:11 | |
| *** hasharAway has quit IRC | 20:28 | |
| *** hashar has joined #openstack-infra | 20:29 | |
| sean-k-mooney | ... my gigabte connect is currently getting 23Mbps down ... | 20:33 |
| sean-k-mooney | i think i may be experincing some network issues | 20:33 |
| *** sshnaidm|bbl has quit IRC | 20:34 | |
| *** sshnaidm|bbl has joined #openstack-infra | 20:35 | |
| fungi | yeah, i don't think opendev's git should be a problem currently, it was earlier before we blocked a bunch of problematic client addresses | 20:38 |
| sean-k-mooney | i am able to clone small repos like devstack or os-vif but nova and neturon dont even get to the | 20:39 |
| sean-k-mooney | remote: Enumerating objects: 112, done. | 20:39 |
| sean-k-mooney | step | 20:39 |
| fungi | ouch | 20:40 |
| fungi | packet loss? | 20:41 |
| *** sean-k-mooney1 has joined #openstack-infra | 20:44 | |
| *** sean-k-mooney has quit IRC | 20:45 | |
| *** sshnaidm|bbl is now known as sshnaidm|afk | 20:46 | |
| sean-k-mooney1 | ok back to 300 after restarting my router good enough | 20:48 |
| fungi | cool, is cloning from opendev.org working okay for you now too? | 20:51 |
| sean-k-mooney1 | so yes opendev.org seams to be working fine. my router is good but its realy not desgined for my network conenction let alone the work im making it do. | 20:51 |
| *** priteau has quit IRC | 20:52 | |
| fungi | heh, a familiar story | 20:52 |
| fungi | my router is openbsd on a pcengines apu2d4, fairly low-resource sbc by modern standards | 20:52 |
| fungi | er, apu1d4 i mean (the 2d4 is a good deal nicer) | 20:53 |
| *** matt_kosut has quit IRC | 20:53 | |
| sean-k-mooney1 | im using one of these https://www.ui.com/edgemax/edgerouter-x/ but i have it doing smart qos (fqdel) + bgp + ipv6 tunneling via hurican eletic | 20:53 |
| *** matt_kosut has joined #openstack-infra | 20:53 | |
| sean-k-mooney1 | but to enable the qos i have to disabel hardware routing | 20:54 |
| sean-k-mooney1 | so it can route at gigabte speed just not with smart queue so im trading off much lower latency vs bandwith | 20:55 |
| fungi | yeah, these are what i'm using: https://www.pcengines.ch/apu1d4.htm | 20:55 |
| fungi | they can manage gigabit throughput between interfaces, though just barely | 20:55 |
| fungi | very well-supported on openbsd though | 20:56 |
| fungi | and i dropped the v6 gre tunnel to he now that i was able to work out my isp's dhcp6-pd expectations | 20:57 |
| *** matt_kosut has quit IRC | 20:57 | |
| sean-k-mooney1 | hum Dual-Core 880 MHz, MIPS1004Kc vs AMD G series T40E, 1 GHz dual Bobcat core and 256 MB DDR3 RAM vs DRAM: 2 or 4 GB DDR3-1066 DRAM | 20:57 |
| fungi | also the gigabit is really only inside the house, i only have ~100mbps to the provider | 20:58 |
| sean-k-mooney1 | i would hope you can get to gigabit in comparison | 20:58 |
| sean-k-mooney1 | fungi: ya so im using HE to give me ipv6 currently | 20:58 |
| fungi | but i can get good throughput between internal interfaces and between different vlans on the same interfaces | 20:58 |
| sean-k-mooney1 | but its been under performing lately | 20:58 |
| sean-k-mooney1 | fungi: i have gigabit FTTH but i dont really have a routher that can handel that | 20:59 |
| clarkb | ubiquiti management software makes me cranky | 21:00 |
| sean-k-mooney1 | so im getting 300-340 ish more or less over wifi | 21:00 |
| *** bandini has quit IRC | 21:00 | |
| sean-k-mooney1 | clarkb: the edgerouter line does nto use there fancy ui | 21:00 |
| clarkb | I have an ubiquiti ap but run similar to fungi for my rputer | 21:00 |
| fungi | clarkb: that's one of the reasons i just use openbsd. command shell and config files. and basically all the network features you could ever want so long as you have supported hardware | 21:00 |
| clarkb | its a 4core amd apu fanless industrial computer | 21:00 |
| sean-k-mooney1 | well if you have any recomendation for something that wont look out of place in my hallway let me know | 21:01 |
| fungi | also these things are small and cheap enough i can buy several and always have a spare preconfigured if something goes wrong | 21:01 |
| fungi | also proper serial console | 21:01 |
| fungi | and "unbrickable" (there's an override bios socket you can plug in a spare rom to reconfigure everything if needed) | 21:02 |
| sean-k-mooney1 | i have been considering geting a pfsense system to replace it but it works well enough for now that im more or less fine with it | 21:02 |
| sean-k-mooney1 | is centos 8 still not supported fully with devstack? | 21:03 |
| fungi | i also have mnie on a direct current ups (basically a specially-designed backup battery with 5v, 9v and 12v outputs) | 21:03 |
| fungi | will run for most of a day during a power outage | 21:04 |
| *** sean-k-mooney1 is now known as sean-k-mooney | 21:06 | |
| fungi | sean-k-mooney: out of place in my hallway let me know | 21:06 |
| fungi | er, that was a terrible series of pointer clicks, sorry | 21:06 |
| fungi | sean-k-mooney: https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-centos-8 | 21:06 |
| *** haleyb has quit IRC | 21:07 | |
| sean-k-mooney | im hitting the normaly disutils issue | 21:07 |
| sean-k-mooney | Cannot uninstall 'PyYAML'. It is a distutils installed project | 21:07 |
| fungi | oh, yeah, you need to not have pyyaml preinstalled from rpm | 21:07 |
| sean-k-mooney | ok ill nuke it and see if it will be happy | 21:08 |
| fungi | we build our own minimal centos-8 images and don't use the default cloud images | 21:08 |
| fungi | but devstack could probably be fixed to remove pyyaml for you when it's doing its package stuff | 21:08 |
| sean-k-mooney | ya i used ironic to deploy a default image becasue my ubuntu cloud images didnt work, im guessing they are missing driver | 21:09 |
| sean-k-mooney | they did not get networking | 21:09 |
| sean-k-mooney | but cloud init pulled in pyyaml | 21:09 |
| sean-k-mooney | given its booted and i can ssh in i dont need it anymore | 21:10 |
| fungi | if you want to test you could grab a recent build from https://nb02.opendev.org/images/ now that your internet is working again | 21:10 |
| fungi | though the images are 9.2gb because we cache recent clones of every project in them | 21:11 |
| sean-k-mooney | if it fails again with random issue i might | 21:11 |
| sean-k-mooney | fungi: its fine i have space | 21:11 |
| sean-k-mooney | ye us glean in stead of cloud-init right | 21:12 |
| sean-k-mooney | to do the sshkey setup | 21:12 |
| sean-k-mooney | with config drive as the source? | 21:12 |
| sean-k-mooney | not 100% sure if that will work with ironic or not | 21:12 |
| sean-k-mooney | but its a dev env so i can delete things if i need too worst case i can just reinstall | 21:13 |
| fungi | right, it would need configdrive, and no i have no idea if it would work with ironic, i was thinking nova/qemu/kvm sorry | 21:13 |
| fungi | though also you could edit the image to bake configuration in or something | 21:13 |
| sean-k-mooney | normally i develop in an openstack vm on kvm but im trying to track down an sriov issue | 21:14 |
| sean-k-mooney | fungi: yep just the dev user element or inject my key | 21:14 |
| sean-k-mooney | our just boot it once an snapshot it then boot the ironic node form that once my key is there | 21:14 |
| *** jberg-dev has quit IRC | 21:14 | |
| fungi | any of those things, yep | 21:15 |
| sean-k-mooney | that siad it should not that hard to get it working | 21:15 |
| *** jbryce has quit IRC | 21:15 | |
| *** _erlon_ has quit IRC | 21:15 | |
| *** jbryce has joined #openstack-infra | 21:15 | |
| *** jberg-dev has joined #openstack-infra | 21:15 | |
| *** _erlon_ has joined #openstack-infra | 21:15 | |
| sean-k-mooney | fungi: is there any reason that i should not add a patch to devstack to unistall pyyaml if it finds it installed ? | 21:16 |
| sean-k-mooney | im mainly wondering why ye created a custom image without it then leave it for devstack to fix | 21:17 |
| sean-k-mooney | was it just left over form when centos did not publish a cloud image? | 21:19 |
| clarkb | I think there are similar fixups for other platforms | 21:20 |
| *** zxiiro has joined #openstack-infra | 21:20 | |
| clarkb | we always build custom images from scratch | 21:20 |
| clarkb | avoids random changes from upstream making updates and allows us to deploy images more quickly than upstream publsihes them | 21:20 |
| sean-k-mooney | clarkb: oh i know that is the case for the ci | 21:20 |
| clarkb | and we can run glean | 21:20 |
| sean-k-mooney | just wondering if anyone would object if i automated this in devstack | 21:20 |
| sean-k-mooney | its still stacking but so far that is the only thing i have had to change | 21:21 |
| clarkb | I don't think anyone will object. Look in tools/fixup* or similar. There is likely arleady a bunch of very similar changes for $platforms | 21:21 |
| sean-k-mooney | like in fixup_stuff.sh | 21:22 |
| clarkb | ya thats the file | 21:22 |
| sean-k-mooney | likeing the name | 21:22 |
| sean-k-mooney | ya we have sudo rm -rf /usr/lib/python3/dist-packages/PyYAML-*.egg-info | 21:22 |
| sean-k-mooney | for ubuntu | 21:22 |
| sean-k-mooney | which will have a similar effect | 21:23 |
| fungi | right, i was trying to say it would probably be a good idea to fix this in devstack so that it can work with official centos cloud images | 21:23 |
| fungi | we aren't going to be using those in zuul our jobs, but devstack targets a lot more environments than just ci | 21:24 |
| sean-k-mooney | if it stacks ill modify my second node with the fix in fixup_fedora which i think also runs for centos/rhel | 21:24 |
| sean-k-mooney | im kind of conficlted baout doing the hack with removing the .egg-info file vs the package removeal | 21:25 |
| sean-k-mooney | the package removal is cleaner | 21:25 |
| sean-k-mooney | and by defualt only cloud init seams to need it | 21:25 |
| fungi | right, the hacky fixups are for things which can't be uninstalled (generally packages considered essential by the distro) | 21:27 |
| sean-k-mooney | is_fedora has a rather broad definition of "fedora" https://github.com/openstack/devstack/blob/master/functions-common#L447-L456 | 21:28 |
| sean-k-mooney | ah you mean where dnf say no | 21:28 |
| fungi | right | 21:28 |
| fungi | if the distro doesn't allow the package to be uninstalled | 21:29 |
| sean-k-mooney | i once forced that to remove pip or someting and endup having yum delete itself | 21:29 |
| sean-k-mooney | reinstalling yum is interesting | 21:29 |
| sean-k-mooney | if i rember correctly i had to use netcat to copy it form another host becasue the system coudl not use scp anymore | 21:30 |
| sean-k-mooney | it was almost one of the "sudo rm -rf /" moments | 21:30 |
| fungi | at least that's pretty easy. just redirect netcat to a file on the receiving system and cat the file to netcat on the sending side | 21:31 |
| fungi | well, "easy" so long as your byte order and encoding are the same on both systems ;) | 21:31 |
| sean-k-mooney | yep it just took a while to figure out what tools were still avaiable as it start unintalling everything based on whatever package we removed | 21:31 |
| *** markvoelker has quit IRC | 21:32 | |
| sean-k-mooney | it was soming like pip or perl that we did not think would break something | 21:32 |
| *** david-lyle has joined #openstack-infra | 21:34 | |
| sean-k-mooney | oh it was python-setuptools we need a new version for pbr i think | 21:34 |
| *** dklyle has quit IRC | 21:35 | |
| fungi | ahh, yeah, without setuptools you likely no longer have pkg_resources and so can't resolve entrypoints and... *foomph* there go some of your fundamental python-based utilities on rh platforms | 21:36 |
| sean-k-mooney | hehe the way the packages were setup python2 depended on python-setuptools and yum depens on python so yep it went boom | 21:37 |
| *** david-lyle is now known as dklyle | 21:37 | |
| *** smarcet has quit IRC | 21:38 | |
| sean-k-mooney | there used to be a seperate pyton-minimal or something ht at python-setuptools depened on but that was a super striped down version | 21:38 |
| fungi | debian does it that way. basically just the interpreter and builtin functions | 21:38 |
| sean-k-mooney | yep no standard lib | 21:39 |
| fungi | then they split the bulk of stdlib into a separate package, and some bits of stdlib into multiple other packages | 21:39 |
| sean-k-mooney | that is when i learn that if your package manager say you will regert this you should proably listen | 21:39 |
| fungi | or have an emergency ramdisk boot handy | 21:40 |
| fungi | and maybe a filesystem snapshot ;) | 21:40 |
| sean-k-mooney | see this is where contianer have a use | 21:41 |
| fungi | except you still need to run the containers somewhere | 21:42 |
| sean-k-mooney | its also where zfs or btrfs filesystem snapshots shine | 21:42 |
| fungi | and your container management layer has to be written in some language and probably has dependencies | 21:42 |
| clarkb | as we discovered with docker-compose | 21:42 |
| fungi | indeed ;) | 21:42 |
| fungi | the problem never entirely goes away, you just hide it with increasing layers of system abstraction and try to forget it's there | 21:43 |
| sean-k-mooney | well the go way of doing it is statically link the worlds | 21:43 |
| sean-k-mooney | which is fine if no one ever rights a security bug | 21:44 |
| fungi | also statically linking libc is painful | 21:45 |
| sean-k-mooney | it depends on the language zig has an intersting way to do it | 21:45 |
| sean-k-mooney | it ships with multiple libc implementaitons | 21:45 |
| sean-k-mooney | and can staticly link it at least for musl but i think others too | 21:46 |
| fungi | huh, i'd never seen ziglang.org until just now | 21:46 |
| sean-k-mooney | it uses libclang as part of its compiler so it can also jsut be used as a c compiler | 21:47 |
| sean-k-mooney | but you can use its cross plathform buidl supprot | 21:47 |
| fungi | https://ziglang.org/#Zig-competes-with-C-instead-of-depending-on-it | 21:47 |
| fungi | interesting approach | 21:47 |
| sean-k-mooney | yep you can still use c form it but it does not plan to require it | 21:48 |
| sean-k-mooney | of the newer langage to emerge in the last 5-10 years i find it one of the more interesting | 21:48 |
| *** smarcet has joined #openstack-infra | 21:49 | |
| sean-k-mooney | its apprcoh to async function is perhaps one of its more interestign feature | 21:49 |
| sean-k-mooney | . | 21:49 |
| *** tobiash has quit IRC | 22:04 | |
| *** tobiash has joined #openstack-infra | 22:06 | |
| *** hashar has quit IRC | 22:07 | |
| sean-k-mooney | hehe ^ | 22:08 |
| *** rlandy is now known as rlandy|afk | 22:15 | |
| *** rfolco has quit IRC | 22:17 | |
| *** piotrowskim has quit IRC | 22:22 | |
| sean-k-mooney | ok if i install python3-mod_wsgi for horizon and remove python3-pyyaml then it devstack stacks fine on the upstream centos 8.1 cloud image | 22:24 |
| sean-k-mooney | technically i think its the centos stream image but i suspect both would work | 22:24 |
| fungi | so not too bad at all | 22:25 |
| sean-k-mooney | ya it was relitivly painless | 22:26 |
| sean-k-mooney | took a minute to realise why httpd was failing but that is just a depency issue | 22:26 |
| *** Lucas_Gray has joined #openstack-infra | 22:32 | |
| *** rfolco has joined #openstack-infra | 22:33 | |
| *** rcernin has joined #openstack-infra | 22:38 | |
| *** tkajinam has joined #openstack-infra | 22:48 | |
| *** rcernin has quit IRC | 22:48 | |
| *** rcernin has joined #openstack-infra | 22:49 | |
| *** tosky has quit IRC | 22:57 | |
| *** smarcet has quit IRC | 22:58 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use a temporary registry with buildx https://review.opendev.org/738517 | 23:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Ignore ansible lint E106 https://review.opendev.org/738716 | 23:19 |
| *** auristor has quit IRC | 23:29 | |
| *** markvoelker has joined #openstack-infra | 23:33 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Ignore ansible lint E106 https://review.opendev.org/738716 | 23:33 |
| *** markvoelker has quit IRC | 23:38 | |
| *** hamalq_ has quit IRC | 23:38 | |
| *** Lucas_Gray has quit IRC | 23:42 | |
| *** markvoelker has joined #openstack-infra | 23:48 | |
| *** markvoelker has quit IRC | 23:53 | |
| *** auristor has joined #openstack-infra | 23:56 | |
| *** Lucas_Gray has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!