Monday, 2020-07-06 appears to have gone bye-bye, and has gummed up the works on bridge00:08
ianwconsole is unresponsive, i'll reboot it00:09
*** ryohayakawa has joined #opendev00:15
ianw#status log rebooted unresponsive elasticsearch05.openstack.org00:22
openstackstatusianw: finished logging00:22
ianwi have also cleared out lots of hung ansible stuff on bridge related to it00:23
ianwERROR: for grafana  b'manifest for opendevorg/grafana:latest not found: manifest unknown: manifest unknown'00:24
ianwmanifest for opendevorg/grafana:latest not found: manifest unknown: manifest unknown00:24
ianwi'm guessing something has changed because the granfa change doesn't seem to like pulling from the intermediate registry any more00:25
ianw... maybe the :latest tag?00:26
ianw ... change_737397_latest00:30
openstackgerritIan Wienand proposed opendev/system-config master: Update grafana-container
clarkbianw: the promote job should add the latest tag00:41
clarkbmaybe that failedfor some reason?00:41
ianwyeah i don't see it @
ianwhrm, maybe it got dropped00:44
openstackgerritIan Wienand proposed opendev/system-config master: Update grafana-container
ianwyeah the job was there, but not added to the run list00:49
*** DSpider has quit IRC01:23
*** xiaolin has joined #opendev01:25
openstackgerritMerged opendev/system-config master: Update grafana-container
openstackgerritMerged openstack/diskimage-builder master: add more python selections to gentoo
*** ykarel|away is now known as ykarel04:13
*** bhagyashris|afk is now known as bhagyashris04:24
*** sgw1 has quit IRC04:28
*** raukadah is now known as chandankumar04:42
*** ysandeep|away is now known as ysandeep04:43
openstackgerritMerged openstack/diskimage-builder master: Revert "Make ipa centos8 job non-voting"
openstackgerritMerged opendev/system-config master: Grafana container deployment
*** marios has joined #opendev05:58
*** sshnaidm|off is now known as sshnaidm|ruck06:17
*** ysandeep is now known as ysandeep|brb06:21
openstackgerritIan Wienand proposed opendev/system-config master: Add host keys to inventory; give host key in launch-node script
*** ysandeep|brb is now known as ysandeep06:34
openstackgerritIan Wienand proposed opendev/system-config master: [wip] add host keys on bridge
*** tosky has joined #opendev07:25
*** dtantsur|afk is now known as dtantsur07:25
openstackgerritIan Wienand proposed opendev/system-config master: [wip] add host keys on bridge
*** fressi has joined #opendev07:57
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** DSpider has joined #opendev08:17
*** marios has quit IRC08:21
*** ysandeep is now known as ysandeep|lunch08:23
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Install yum and yum-utils on Red Hat family hosts
*** persia has quit IRC08:44
*** persia has joined #opendev08:45
*** bhagyashris is now known as bhagyashris|afk08:50
*** hashar has joined #opendev08:53
*** ysandeep|lunch is now known as ysandeep08:56
*** marios has joined #opendev09:16
*** bhagyashris|afk is now known as bhagyashris09:19
openstackgerritIan Wienand proposed opendev/system-config master: [wip] add host keys on bridge
openstackgerritRiccardo Pittau proposed openstack/diskimage-builder master: Convert multi line if statement to case
*** tkajinam has quit IRC10:37
openstackgerritRadosÅ‚aw Piliszek proposed openstack/diskimage-builder master: [CI] Do not run functionality tests on docs change
sshnaidm|ruckfolks, please merge it prevents retry_limits in jobs10:51
openstackgerrityatin proposed zuul/zuul-jobs master: [DNM] Test task in a role
*** lpetrut has joined #opendev11:26
openstackgerritMerged openstack/diskimage-builder master: Install yum and yum-utils on Red Hat family hosts
*** ryohayakawa has quit IRC12:08
*** priteau has joined #opendev12:21
openstackgerritRafael Folco proposed openstack/diskimage-builder master: Enable py3 on dib centos/rhel 7
*** ysandeep is now known as ysandeep|afk12:38
tristanCclarkb: it seems like you have been looking into persistent-firewall already, could you please have a look at the proposed fix here:
*** ysandeep|afk is now known as ysandeep12:56
openstackgerritMerged openstack/diskimage-builder master: Stop bringing the test environment into the mocks
*** redrobot has joined #opendev13:34
*** lpetrut has quit IRC14:32
openstackgerritMonty Taylor proposed opendev/system-config master: Build multi-arch python-base/python-builder
mordredcorvus: with your updates to multi-arch images - I think we should be in good shape to land that ^^ now yeah?14:45
*** ykarel is now known as ykarel|away14:46
*** mlavalle has joined #opendev14:52
*** factor has joined #opendev14:57
*** rajinir_ has joined #opendev15:02
*** hashar has quit IRC15:06
mordredparking restrictions have been suspended in new orleans - this means it's about to be very wet15:11
*** ysandeep is now known as ysandeep|away15:22
*** sshnaidm|ruck is now known as sshnaidm|mtg15:24
*** factor has quit IRC15:25
*** icarusfactor has joined #opendev15:25
*** sgw1 has joined #opendev15:27
corvusmordred: +2! :)15:44
corvustristanC: (cc clarkb, ianw, fungi) i've also been working with mwhahaha on trying to find the root cause of the iptables issue; we think it may be an errno being returned by the ansible command plugin, and mwhahaha suspects that it's a return value from a fcntl call, but we're still trying to catch it in the act.15:49
corvussshnaidm|mtg: ^ fyi15:50
corvuson friday we were using 739061 to try to catch the issue in action15:51
*** marios is now known as marios|out15:52
*** hashar has joined #opendev15:52
*** marios|out has quit IRC15:54
*** slittle1 has quit IRC16:12
*** priteau has quit IRC16:14
*** sgw1 has quit IRC16:21
tristanCcorvus: it sounds like 739061 would be helpfull in anycase, though i'm not entirely sure why retrying would work16:22
*** sgw1 has joined #opendev16:23
*** sgw1 has quit IRC16:31
*** sgw1 has joined #opendev16:34
openstackgerritGhanshyam Mann proposed openstack/project-config master: Add "Review-Priority" label to qa projects
openstackgerritGhanshyam Mann proposed openstack/project-config master: Add "Review-Priority" label to qa projects
*** dtantsur is now known as dtantsur|afk16:54
*** slittle1 has joined #opendev17:06
corvustristanC: yeah, i don't know that we've seen retrying actually work, but it seems plausible that it could; certainly unlikely to hurt.  i'm not sure if folks have set depends-on to that yet.17:09
*** priteau has joined #opendev17:22
*** hashar has quit IRC17:22
openstackgerritMerged zuul/zuul-jobs master: Make persist-iptables more robust
*** priteau has quit IRC18:07
openstackgerritMerged openstack/diskimage-builder master: add musl profile to gentoo
openstackgerritMerged openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions
*** xiaolin has quit IRC18:56
*** sshnaidm|mtg is now known as sshnaidm|ruck19:01
*** icarusfactor has quit IRC19:16
*** shtepanie has joined #opendev19:18
mnaserso this might sound a little wild, but.. is there any projects that do cross-tenant third-party-ci inside opendev?19:26
mnaserfor example, if the vexxhost tenant wants to do 3pci for openstack/keystone so that they get a signal if they break vexxhost/openstack-operator19:26
mnaseri mean, i could totally do normal 3pci, but it seems like it might more sense in the same context/space/place19:27
mordredmnaser: this is a use case that folks had a while ago (before zuul v3) and I believe is why jhesketh added multi-connection support. my brain is having a tough time paging in current context19:35
mordredcorvus: ^^19:35
mnasermordred: i mean, i think it probably is very likely possible, though i guess it may require vexxhost getting its own pipeline config which maybe a little bit more than opendev wants to deal with19:38
mnaser(understandable-y so, but if it seems like an idea folks are behind, i'm happy to do it for future tenants reference)19:38
corvusi think the tenant config part is less of an issue than the connection part.  it would need a new gerrit connection19:41
corvus(in order to leave a vote as a different user)19:42
corvusit would definitely be extra work/complexity/etc for the opendev admin team, though probably less so than the total human cost of vexxhost running another zuul just for this.  :)19:43
mordredcorvus: nod - so that's still the situation - we'd want a second zuul account that opendev's zuul would use for that purpose19:49
mnasercorvus: i mean, we can easily start doing 3pci but the idea was that if it made sense to 'pave the way' for future possibilities of tenants interacting with each other19:49
mnaserif that's not a use case we're looking to handle at some point, then yeah, it could be interesting to check it out19:50
mnaseri'm thinking especially since maybe if airship has its own tenant and wants to vote on i dunno, openstack-helm for example19:50
mnasermordred: perhaps one single extra opendev connection that would do 3pci on all other tenants.. if that makes sense -- but that perhaps doubles the work the zuul-scheduler needs to do19:51
fungialso every time this comes up, the idea of having reporting via multiple user accounts seems like a workaround for our (more accurately i guess gerrit's) data model not being able to adequately represent the information we want to convey20:06
fungiare there aspects of the new checks api which would allow us to do this with a single account?20:06
fungi(assuming zuul had a different means of grouping vexxhost vs openstack builds for the same items and pipelines)20:07
fungilike if we had some way of indicating that a given buildset came from the gating tenant vs merely a non-blocking advisory tenant20:10
mnaserfungi: well i mean, technically, we _could_ have zuul report twice, one for another pipeline called vexxhost-3pci and it would enqueue at once? similar to our arm64 pipelnie today?20:10
corvusfungi: i don't think checks is advantageous here; the main issue is that a single column gets a single vote per user (checks would be the same, a single check gets a single result)20:10
corvusyeah, a key think about arm64 is that it doesn't vote20:11
fungiin the current plan is a check mapped 1:1 to a pipeline/buildset? (i gather subchecks are the 1:1 mapping to an individual job/build)?20:12
mnaseri think my initial intention was non-voting external 3pci too20:12
corvusfungi: yes (pipeline)20:12
corvusmnaser: if you're not concerned with leaving a vote, then a second account isn't necessary20:13
corvusi am told, however, that "people ignore results without votes"20:13
fungii'm told that people ignore third-party ci results with votes too20:13
fungihence tripleo folks asking last week whether they could make their rdo ci block tripleo changes from merging20:14
corvusmnaser: maybe if you want to pursue this, that's a good way to start?  fewer technical hurdles, mostly policy/procedural ones, and could potentially be upgraded to voting later with addition of another account.20:15
mnasercorvus, fungi: heh, well, yeah, i guess it can be ignored.  i'm hoping to try and ask nicely to 'watch' for the signal20:17
mnaseri guess my + side is that these jobs will probably finish faster than the normal check pipeline so maybe eyes will be on them :)20:18
*** hashar has joined #opendev20:54
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add openstack/devstack-plugin-ceph in vexxhost tenant
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add openstack/devstack-plugin-ceph in vexxhost tenant
*** xiaolin has joined #opendev21:03
*** xiaolin has quit IRC21:11
donnydsoooo I was on-site today and just got back... my air handler is pretty much done.. going to need to turn OE off for a while.. not sure when I can get it fixed21:13
openstackgerritDonny Davis proposed openstack/project-config master: Air Handler is failed - turn OE off for now
mnaserdonnyd: sorry to hear21:15
donnydmnaser:  so goes being a little guy I guess21:16
donnydI have kept it chugging along for the last year or so when it crapped the bed last summer21:17
mnaserdonnyd: appreciate every bit of providing infra though, every build minute helps :)21:17
donnydI will now need to come up with a plan to find/build a new one - repairing the old one isn't going to cut it anymore I guess21:18
donnydI am happy to contribute where and when I can.. this community is important to me21:18
fungidonnyd: your help is invaluable and MUCH appreciated!21:20
fungihaving run both commercial and home data centers, i can agree that cooling is probably tied with power distribution for the hardest problems to solve21:21
donnydit has been working ok till this morning.. but it finally just died21:26
donnydI will let you guys know when its back up21:26
fungigood luck! i've been struggling with our residential unit here too, we just paid rather too much to have it all replaced a few weeks ago and it's still not quite working how we'd like21:28
donnydI have a 15K bill to get it fixed by a professional - which unfortunately is not in the cards at the moment21:29
donnydTHankfully the whole system isn't failed - just the part that powers the D/C is unable to cope21:30
fungiyep, not good. that's roughly what the replacement for the entire hvac for our house ran (including rerunning ducts)21:31
openstackgerritMerged openstack/project-config master: Air Handler is failed - turn OE off for now
*** Eighth_Doctor is now known as Eleventh_Doctor22:16
*** Eleventh_Doctor is now known as Eighth_Doctor22:29
*** tkajinam has joined #opendev22:42
*** shtepanie has quit IRC22:48
*** tosky has quit IRC22:49 was out of space, which was causing the base job to fail22:54
ianwtrack_upstream.log was full of failure to connect messages which seems to be the culprit, i have removed it22:54
fungii thought we merged a change to skip review-test in those jobs?22:55
ianwumm, it's full again?  something screwy going on here23:00
ianw# docker ps | wc -l23:01
ianwthat doesn't seem right23:02
*** mlavalle has quit IRC23:04
ianwwell i've stopped all 210 containers and some space has freed up.  clearly something got in a loop23:05
ianwit doesn't seem the container starts on boot.  i'll just leave it for now; it is at least not going to affect the base playbook run now23:08
ianwclarkb: i know you're not around but is there a meeting tomorrow?23:09
clarkbianw: I havemt seen anyone volunteer to host it yet23:10
ianwclarkb: ok, i can, just to keep something in the record.  i will send an agenda23:12
fungii'll be around, could be a good opportunity to talk about what to do with the iptables rules for china unicom23:24
*** factor has joined #opendev23:31
*** hrw has quit IRC23:44
*** hrw has joined #opendev23:44
*** hashar has quit IRC23:56

Generated by 2.17.2 by Marius Gedminas - find it at!