| *** opendevtest <opendevtest!~limnoria@104.239.144.232> has joined #opendev | 01:15 | |
| *** Guest1653 <Guest1653!~limnoria@104.239.144.232> has joined #opendev | 01:25 | |
| opendevreview | melanie witt proposed opendev/jeepyb master: Convert update_blueprint to use the Gerrit REST API https://review.opendev.org/c/opendev/jeepyb/+/795912 | 01:27 |
|---|---|---|
| ianw | oh, it looks like i have not setup limnoria correctly to identify to nickserv | 01:28 |
| ianw | !plugins | 01:30 |
| Guest1653 | ianw: Error: "plugins" is not a valid command. | 01:30 |
| *** opendevmeet` <opendevmeet`!~limnoria@104.239.144.232> has joined #opendev | 01:32 | |
| opendevreview | melanie witt proposed opendev/system-config master: Re-enable update_blueprint for patchset-created https://review.opendev.org/c/opendev/system-config/+/795914 | 01:34 |
| *** opendevmeet` <opendevmeet`!~limnoria@104.239.144.232> has joined #opendev | 01:39 | |
| *** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #opendev | 02:02 | |
| corvus | #status log restarted all of zuul on commit dd45f931b62ef6a5362e39bdb56ee203b74e1381 (4.5.0 +1) | 02:02 |
| opendevstatus | corvus: finished logging | 02:02 |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria: production fixes https://review.opendev.org/c/opendev/system-config/+/795917 | 02:02 |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has joined #opendev | 02:07 | |
| corvus | re-enqueing | 02:08 |
| corvus | ianw: ^ that may need a recheck | 02:09 |
| *** boistordu <boistordu!~boistordu@0002bdcc.user.oftc.net> has joined #opendev | 02:11 | |
| *** boistordu_ex <boistordu_ex!~boistordu@0002bdcc.user.oftc.net> has quit IRC (Ping timeout: 480 seconds) | 02:17 | |
| corvus | re-enqueue is done | 02:20 |
| *** ysandeep|out is now known as ysandeep | 02:34 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria: production fixes https://review.opendev.org/c/opendev/system-config/+/795917 | 02:52 |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has quit IRC (Ping timeout: 480 seconds) | 03:04 | |
| *** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #opendev | 03:04 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria: production fixes https://review.opendev.org/c/opendev/system-config/+/795917 | 03:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: add mariadb_container option https://review.opendev.org/c/opendev/system-config/+/775961 | 03:32 |
| opendevreview | Ian Wienand proposed opendev/system-config master: review02 : switch reviewdb to mariadb_container type https://review.opendev.org/c/opendev/system-config/+/795192 | 03:32 |
| *** redrobot <redrobot!~redrobot@108-84-79-198.lightspeed.snantx.sbcglobal.net> has quit IRC (Remote host closed the connection) | 03:40 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: static: enable SSLProxyEngine for meetings https://review.opendev.org/c/opendev/system-config/+/795920 | 03:43 |
| *** ysandeep <ysandeep!~sandy@202.173.126.121> has quit IRC (Ping timeout: 480 seconds) | 04:07 | |
| *** ykarel <ykarel!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has joined #opendev | 04:22 | |
| opendevreview | Merged opendev/system-config master: limnoria: production fixes https://review.opendev.org/c/opendev/system-config/+/795917 | 04:25 |
| *** ricolin_ <ricolin_!~ricolin@118.150.144.205> has joined #opendev | 04:31 | |
| *** ricolin <ricolin!~ricolin@118.150.144.205> has quit IRC (Ping timeout: 480 seconds) | 04:35 | |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has joined #opendev | 04:39 | |
| *** ysandeep <ysandeep!~sandy@202.173.126.121> has joined #opendev | 04:50 | |
| opendevreview | Merged opendev/system-config master: static: enable SSLProxyEngine for meetings https://review.opendev.org/c/opendev/system-config/+/795920 | 05:02 |
| ianw | testing | 05:20 |
| *** marios <marios!~marios@62-171-24.netrun.cytanet.com.cy> has joined #opendev | 05:37 | |
| *** marios is now known as marios|ruck | 05:41 | |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has quit IRC (Ping timeout: 480 seconds) | 06:17 | |
| ianw | https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2021-06-11.log.html looks good | 06:20 |
| ianw | i've tested a few eavesdrop links and they all correctly bounce eavesdrop01.openstack.org -> meetings.opendev.org -> (proxy) -> eavesdrop01.opendev.org | 06:22 |
| ianw | i think we can tick this one off | 06:22 |
| ianw | #status log meetbot/logging now running from limnoria on eavesdrop01.opendev.org | 06:23 |
| opendevstatus | ianw: finished logging | 06:23 |
| *** ralonsoh <ralonsoh!~ralonsoh@36.red-79-150-231.dynamicip.rima-tde.net> has joined #opendev | 06:27 | |
| *** amoralej <amoralej!~amoralej@153.red-80-26-161.dynamicip.rima-tde.net> has joined #opendev | 06:43 | |
| *** hashar <hashar!~hashar@hashar.user.oftc.net> has joined #opendev | 07:13 | |
| *** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has joined #opendev | 07:14 | |
| *** rpittau|afk is now known as rpittau | 07:17 | |
| *** andrewbonney <andrewbonney!uid417545@id-417545.highgate.irccloud.com> has joined #opendev | 07:27 | |
| *** jpena|off is now known as jpena | 07:33 | |
| opendevreview | Merged opendev/system-config master: Cleanup eavesdrop puppet references https://review.opendev.org/c/opendev/system-config/+/795014 | 07:45 |
| opendevreview | Merged opendev/system-config master: Run statusbot from eavesdrop01.opendev.org https://review.opendev.org/c/opendev/system-config/+/795213 | 07:46 |
| *** lucasagomes <lucasagomes!~lucasagom@89.100.20.18> has joined #opendev | 07:56 | |
| *** ysandeep is now known as ysandeep|lunch | 08:06 | |
| *** opendevstatus is now known as Guest1684 | 08:09 | |
| *** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has quit IRC (Ping timeout: 480 seconds) | 08:13 | |
| *** mgoddard- <mgoddard-!~mgoddard@238.240.125.91.dyn.plus.net> has joined #opendev | 08:14 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Move statusbot channels out of hiera https://review.opendev.org/c/opendev/system-config/+/795958 | 08:16 |
| *** mgoddard <mgoddard!~mgoddard@187.240.125.91.dyn.plus.net> has quit IRC (Ping timeout: 480 seconds) | 08:18 | |
| *** mgoddard- is now known as mgoddard | 08:18 | |
| frickler | ianw: this doesn't look correct to me: 08:09 -!- opendevstatus is now known as Guest1684 | 08:21 |
| *** sshnaidm is now known as sshnaidm|afk | 08:24 | |
| *** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has joined #opendev | 08:24 | |
| *** ykarel is now known as ykarel|lunch | 08:31 | |
| *** Guest1685 <Guest1685!~limnoria@104.239.144.232> has joined #opendev | 08:40 | |
| *** opendevstatus_ <opendevstatus_!~opendevst@104.130.70.91> has joined #opendev | 08:45 | |
| *** opendevstatus_ is now known as opendevstatus__ | 08:47 | |
| *** opendevstatus__ is now known as opendevstatus___ | 08:47 | |
| *** opendevstatus___ is now known as opendevstatus____ | 08:47 | |
| *** opendevstatus____ is now known as opendevstatus_____ | 08:48 | |
| *** opendevstatus_____ is now known as opendevstatus______ | 08:48 | |
| ianw | frickler: sorry still working on opendevstatus atm | 08:57 |
| *** opendevstatus______ <opendevstatus______!~opendevst@104.130.70.91> has quit IRC (Ping timeout: 480 seconds) | 08:58 | |
| *** Guest1684 <Guest1684!~opendevst@eavesdrop01.openstack.org> has quit IRC (Remote host closed the connection) | 08:58 | |
| *** ysandeep|lunch <ysandeep|lunch!~sandy@202.173.126.121> has quit IRC (Ping timeout: 480 seconds) | 08:59 | |
| ianw | i've just killed the statusbot running in screen on eavesdrop01.openstack.org. the service is running on eavesdrop01.opendev.org now, and i'm just waiting on 795958 to give us the channel config | 09:00 |
| ianw | sorry that above was the two bots fighting for the name | 09:00 |
| *** opendevstatus_ <opendevstatus_!~opendevst@158.69.72.85> has joined #opendev | 09:13 | |
| *** opendevstatus_ is now known as opendevstatus__ | 09:16 | |
| *** opendevstatus__ is now known as opendevstatus___ | 09:16 | |
| *** opendevstatus___ is now known as opendevstatus____ | 09:16 | |
| *** opendevstatus____ is now known as opendevstatus_____ | 09:16 | |
| *** opendevstatus_____ is now known as opendevstatus______ | 09:16 | |
| opendevreview | Merged opendev/system-config master: Move statusbot channels out of hiera https://review.opendev.org/c/opendev/system-config/+/795958 | 09:23 |
| *** opendevstatus______ <opendevstatus______!~opendevst@158.69.72.85> has quit IRC (Ping timeout: 480 seconds) | 09:25 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #opendev | 09:31 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection) | 09:31 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #opendev | 09:31 | |
| ianw | #status log statusbot running on eavesdrop01.opendev.org | 09:32 |
| opendevstatus | ianw: finished logging | 09:34 |
| ianw | thankyou statusbot | 09:34 |
| ianw | that is visible on https://wiki.openstack.org/wiki/Infrastructure_Status | 09:35 |
| *** ysandeep|lunch <ysandeep|lunch!~sandy@202.173.126.121> has joined #opendev | 09:41 | |
| *** ysandeep|lunch is now known as ysandeep | 09:47 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria: don't log channel join/parts https://review.opendev.org/c/opendev/system-config/+/795972 | 09:47 |
| ianw | i believe meetbot and statusbot are now fully deployed on eavesdrop01.opendev.org | 09:49 |
| ianw | i think i will shutdown eavesdrop01.openstack.org to avoid any confusion. this leaves ptg still todo, but we know about that | 09:50 |
| *** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #opendev | 09:55 | |
| *** opendevmeet is now known as Guest1688 | 09:56 | |
| *** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #opendev | 09:57 | |
| *** opendevmeet is now known as Guest1689 | 09:57 | |
| *** Guest1690 <Guest1690!~limnoria@104.239.144.232> has joined #opendev | 10:03 | |
| *** opendevmeet` <opendevmeet`!~limnoria@104.239.144.232> has joined #opendev | 10:16 | |
| *** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #opendev | 11:15 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria: fix nicks syntax https://review.opendev.org/c/opendev/system-config/+/795988 | 11:15 |
| *** Guest1712 <Guest1712!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection) | 11:17 | |
| *** Guest1713 <Guest1713!~opendevst@149.202.169.13> has quit IRC (Ping timeout: 480 seconds) | 11:18 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #opendev | 11:18 | |
| ianw | when i was debugging connecting as opendevmeet (which can't really do in the gate) i manually fixed the config file correctly. but then i committed the typo ^, so when ansible applied it, it put in the broken config | 11:21 |
| ianw | that's why it was working then stopped | 11:22 |
| ianw | anyway, once this little stack of fixes merges, i think we're all good | 11:22 |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** opendevstatus_ <opendevstatus_!~opendevst@104.130.219.52> has joined #opendev | 11:40 | |
| *** opendevstatus_ is now known as opendevstatus__ | 11:43 | |
| *** opendevstatus__ is now known as opendevstatus___ | 11:43 | |
| *** opendevstatus___ is now known as opendevstatus____ | 11:43 | |
| *** opendevstatus____ is now known as opendevstatus_____ | 11:43 | |
| *** opendevstatus_____ is now known as opendevstatus______ | 11:43 | |
| *** opendevstatus______ <opendevstatus______!~opendevst@104.130.219.52> has quit IRC (Ping timeout: 480 seconds) | 11:51 | |
| *** ykarel is now known as ykarel|afk | 12:07 | |
| *** whayutin <whayutin!~weshay|ru@c-73-229-75-146.hsd1.co.comcast.net> has joined #opendev | 12:09 | |
| *** opendevstatus_ <opendevstatus_!~opendevst@104.130.219.164> has joined #opendev | 12:12 | |
| *** opendevstatus_ is now known as opendevstatus__ | 12:14 | |
| *** opendevstatus__ is now known as opendevstatus___ | 12:15 | |
| *** opendevstatus___ is now known as opendevstatus____ | 12:15 | |
| *** opendevstatus____ is now known as opendevstatus_____ | 12:15 | |
| *** opendevstatus_____ is now known as opendevstatus______ | 12:15 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection) | 12:18 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #opendev | 12:18 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection) | 12:22 | |
| *** opendevstatus______ <opendevstatus______!~opendevst@104.130.219.164> has quit IRC (Ping timeout: 480 seconds) | 12:24 | |
| *** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #opendev | 12:26 | |
| *** ysandeep is now known as ysandeep|mtg | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: statusbot: don't use opendevstatus name in testing https://review.opendev.org/c/opendev/system-config/+/795998 | 12:35 |
| *** opendevstatus_ <opendevstatus_!~opendevst@213.32.72.249> has joined #opendev | 12:41 | |
| *** opendevstatus_ is now known as opendevstatus__ | 12:44 | |
| *** opendevstatus__ is now known as opendevstatus___ | 12:44 | |
| *** opendevstatus___ is now known as opendevstatus____ | 12:44 | |
| *** opendevstatus____ is now known as opendevstatus_____ | 12:44 | |
| *** opendevstatus_____ is now known as opendevstatus______ | 12:44 | |
| ianw | ^ will stop this happening; it's trying to connect during testing | 12:45 |
| *** amoralej is now known as amoralej|lunch | 12:46 | |
| *** opendevstatus______ <opendevstatus______!~opendevst@213.32.72.249> has quit IRC (Ping timeout: 480 seconds) | 12:53 | |
| *** ykarel|afk is now known as ykarel | 12:57 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: statusbot: don't use opendevstatus name in testing https://review.opendev.org/c/opendev/system-config/+/795998 | 12:59 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Add gmann to IRC accessbot https://review.opendev.org/c/openstack/project-config/+/795986 | 13:07 |
| *** opendevstatus_ <opendevstatus_!~opendevst@104.130.26.53> has joined #opendev | 13:12 | |
| *** opendevstatus_ is now known as opendevstatus__ | 13:15 | |
| *** opendevstatus__ is now known as opendevstatus___ | 13:15 | |
| *** opendevstatus___ is now known as opendevstatus____ | 13:15 | |
| *** opendevstatus____ is now known as opendevstatus_____ | 13:15 | |
| *** opendevstatus_____ is now known as opendevstatus______ | 13:15 | |
| opendevreview | Merged opendev/system-config master: limnoria: fix nicks syntax https://review.opendev.org/c/opendev/system-config/+/795988 | 13:17 |
| *** CeeMac <CeeMac!uid366483@id-366483.brockwell.irccloud.com> has quit IRC (Quit: Connection closed for inactivity) | 13:19 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Update eavesdrop deploy job https://review.opendev.org/c/opendev/system-config/+/796006 | 13:24 |
| *** opendevstatus______ <opendevstatus______!~opendevst@104.130.26.53> has quit IRC (Ping timeout: 480 seconds) | 13:24 | |
| *** amoralej|lunch is now known as amoralej | 13:28 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: statusbot: don't prefix with extra # for testing https://review.opendev.org/c/opendev/system-config/+/796009 | 13:32 |
| *** artom_ <artom_!~artom@205.233.59.73> has quit IRC (Remote host closed the connection) | 13:35 | |
| *** artom_ <artom_!~artom@205.233.59.73> has joined #opendev | 13:35 | |
| ianw | ok, all config changes rolled out, the meetbot and statusbot containers should be happy and in a steady state | 13:35 |
| ianw | i'm going to turn in now | 13:35 |
| *** ysandeep|mtg is now known as ysandeep | 13:37 | |
| *** artom <artom!~artom@205.233.59.73> has joined #opendev | 13:40 | |
| opendevreview | Danni Shi proposed openstack/diskimage-builder master: Add a keylime-agent element and a tpm-emulator element https://review.opendev.org/c/openstack/diskimage-builder/+/789601 | 13:40 |
| *** ysandeep is now known as ysandeep|out | 13:45 | |
| *** artom_ <artom_!~artom@205.233.59.73> has quit IRC (Ping timeout: 480 seconds) | 13:47 | |
| *** artom <artom!~artom@205.233.59.73> has quit IRC (Remote host closed the connection) | 13:50 | |
| *** artom <artom!~artom@205.233.59.73> has joined #opendev | 13:51 | |
| opendevreview | Merged opendev/system-config master: statusbot: don't use opendevstatus name in testing https://review.opendev.org/c/opendev/system-config/+/795998 | 14:02 |
| *** ralonsoh <ralonsoh!~ralonsoh@36.red-79-150-231.dynamicip.rima-tde.net> has quit IRC (Quit: Leaving) | 14:14 | |
| *** ralonsoh <ralonsoh!~ralonsoh@36.red-79-150-231.dynamicip.rima-tde.net> has joined #opendev | 14:16 | |
| *** artom <artom!~artom@205.233.59.73> has quit IRC (Quit: Leaving) | 14:37 | |
| *** artom <artom!~artom@205.233.59.73> has joined #opendev | 14:38 | |
| *** ysandeep|out <ysandeep|out!~sandy@202.173.126.121> has quit IRC (Ping timeout: 480 seconds) | 14:38 | |
| *** dklyle <dklyle!~dklyle@134.134.139.72> has joined #opendev | 14:51 | |
| *** david-lyle <david-lyle!~dklyle@jfdmzpr05-ext.jf.intel.com> has quit IRC (Remote host closed the connection) | 14:57 | |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has joined #opendev | 15:06 | |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has quit IRC (Ping timeout: 480 seconds) | 15:16 | |
| clarkb | ianw: thank you for taking care of that! | 15:24 |
| *** hashar <hashar!~hashar@hashar.user.oftc.net> has quit IRC (Quit: I am a virus. Please copy paste me in your /quit message to help me propagate) | 15:29 | |
| *** marios|ruck is now known as marios|out | 15:37 | |
| *** ykarel is now known as ykarel|away | 15:44 | |
| *** rpittau is now known as rpittau|afk | 15:47 | |
| *** odyssey4me <odyssey4me!~odyssey4m@host31-51-109-193.range31-51.btcentralplus.com> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…) | 15:50 | |
| *** ykarel|away <ykarel|away!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has quit IRC (Ping timeout: 480 seconds) | 15:56 | |
| *** lucasagomes <lucasagomes!~lucasagom@89.100.20.18> has quit IRC (Quit: Leaving) | 15:56 | |
| *** ysandeep <ysandeep!~sandy@202.173.126.240> has joined #opendev | 16:04 | |
| *** ysandeep <ysandeep!~sandy@202.173.126.240> has quit IRC () | 16:04 | |
| *** timburke <timburke!~timburke@2601:645:c480:3660:d4f3:a3b4:736b:710d> has joined #opendev | 16:11 | |
| *** marios|out <marios|out!~marios@62-171-24.netrun.cytanet.com.cy> has quit IRC (Ping timeout: 480 seconds) | 16:11 | |
| clarkb | fungi: ianw mentioned that logan- had responded and we might want to consider https://review.opendev.org/c/openstack/project-config/+/794406 during US hours for better overlap | 16:15 |
| clarkb | fungi: any thoughts on landing that now? I'll be around today if we need to disable it again | 16:16 |
| *** jpena is now known as jpena|off | 16:17 | |
| fungi | looks like it's still workflow -1... will need to revise the change or delete ianw's vote from it | 16:18 |
| clarkb | ya I guess we can't as easily click the 'x' to remove the WIP vote anymore | 16:18 |
| clarkb | a new ps is probably easiest | 16:18 |
| fungi | or temporarily elevating account perms | 16:19 |
| *** amoralej is now known as amoralej|off | 16:23 | |
| *** amoralej|off <amoralej|off!~amoralej@153.red-80-26-161.dynamicip.rima-tde.net> has quit IRC (Quit: Leaving) | 16:24 | |
| fungi | i have temporarily elevated my perms to delete ianw's workflow -1 from 794406 and am approving it | 16:36 |
| clarkb | cool | 16:36 |
| opendevreview | Merged openstack/project-config master: Revert "Revert "Revert "Disable limestone due to mirror issues""" https://review.opendev.org/c/openstack/project-config/+/794406 | 16:46 |
| *** andrewbonney <andrewbonney!uid417545@id-417545.highgate.irccloud.com> has quit IRC (Quit: Connection closed for inactivity) | 17:09 | |
| *** ralonsoh <ralonsoh!~ralonsoh@36.red-79-150-231.dynamicip.rima-tde.net> has quit IRC (Quit: Leaving) | 17:19 | |
| clarkb | fungi: let me get a few commands stashed in a text document really quickly before I start | 17:41 |
| *** slittle1 <slittle1!~slittle@108.162.140.52> has quit IRC (Read error: Connection reset by peer) | 17:42 | |
| * fungi is standing by to test mirrors | 17:42 | |
| *** slittle1 <slittle1!~slittle@108.162.140.52> has joined #opendev | 17:43 | |
| clarkb | #status Notice Zuul is being restarted for server reboots | 17:45 |
| opendevstatus | clarkb: sending notice | 17:45 |
| -opendevstatus- NOTICE: Zuul is being restarted for server reboots | 17:46 | |
| clarkb | fungi: zuul is stopping now | 17:47 |
| fungi | awesome | 17:48 |
| opendevstatus | clarkb: finished sending notice | 17:48 |
| clarkb | zm01 and zm02 failed to reboot because they are not accessible via ssh according to ansible | 17:48 |
| fungi | good to see opendevstatus is still able to send notifications | 17:49 |
| fungi | so they were already down? | 17:49 |
| *** CeeMac <CeeMac!uid366483@id-366483.brockwell.irccloud.com> has joined #opendev | 17:49 | |
| clarkb | not sure yet. ze01 04 and 05 show similar | 17:49 |
| clarkb | I think thats an ansible behavior | 17:50 |
| clarkb | the reboot closed the ssh connection before ansible was done with it | 17:50 |
| clarkb | at least zm01 and zm02 report small uptimes | 17:50 |
| clarkb | I'll proceed with the mirror reboots and check the ze's that errored after | 17:50 |
| fungi | ahh, okay | 17:53 |
| clarkb | fungi: the 6 focal mirrors are done, but they all produced the same error. Cna you check their uptimes when you check their afs as well? | 17:53 |
| clarkb | I'm checking zuul servers now | 17:53 |
| fungi | yep, testing the mirrors now | 17:54 |
| clarkb | I'm checking uptimes with ansible too | 17:55 |
| clarkb | uptimes all lgtm. Let me know if you think afs is happy then I can run the zuul start playbook | 17:57 |
| fungi | i can't reach https://mirror.regionone.osuosl.opendev.org/ | 17:58 |
| fungi | i also seem to be timing out on https://mirror.regionone.linaro-us.opendev.org | 17:58 |
| clarkb | osuosl's mirror shows the same dmesg problem that limestone did | 17:59 |
| clarkb | or maybe its a different error. /me checks linaro next | 17:59 |
| fungi | it's possible my wireless modem being ipv4-only is presenting some v6 connectivity problems for them | 17:59 |
| fungi | okay, i did finally get a response from https://mirror.regionone.linaro-us.opendev.org/ | 18:00 |
| clarkb | linaro doesn't show the oops/protection fault in dmesg | 18:00 |
| clarkb | it is having a hard time with ls in /afs/openstack.org though | 18:00 |
| fungi | looks like mirror.regionone.osuosl.opendev.org is v4-only | 18:00 |
| clarkb | fungi: did you want to try cleanign out the openafs cache on the osuosl mirror? | 18:01 |
| fungi | so not a local v6 problem on my end | 18:01 |
| clarkb | yes I believe we don't have ipv6 there yet | 18:01 |
| fungi | yeah, i'll give that a shot | 18:01 |
| clarkb | linaro looks happy now I can ls in there. I think if you're happy with osuosl after cleaning the cache up we're ready to start zuul | 18:02 |
| fungi | removing /var/cache/openafs/* on mirror01.regionone.osuosl.opendev.org now | 18:03 |
| fungi | it's taking a few minutes | 18:03 |
| clarkb | looks empty now? | 18:04 |
| clarkb | dont' worry about my ssh session feel free to reboot again when you are ready | 18:04 |
| fungi | okay, deletion complete, rebooting it now | 18:04 |
| *** odyssey4me <odyssey4me!~odyssey4m@rdng-28-b2-v4wan-161903-cust132.vm39.cable.virginm.net> has joined #opendev | 18:05 | |
| *** ykarel|away <ykarel|away!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has joined #opendev | 18:05 | |
| clarkb | shoudl I go ahead and start zuul since that is the last remaining sad server? | 18:07 |
| clarkb | and zuul will take some time anyway? | 18:07 |
| clarkb | it isn't pinging :/ | 18:07 |
| fungi | yeah, go for it. worse case we emergency turn down this provider until we get the mirror for it back on track | 18:08 |
| clarkb | ok starting zuul now | 18:08 |
| fungi | i'm still waiting for it to boot | 18:08 |
| clarkb | fungi: I wonder if we'll need to ask nova to hard reboot that osuosl mirror | 18:10 |
| clarkb | I wonder if the kernel isn't unloading openafs properly or some similar unit is failing to stop making it slow | 18:10 |
| fungi | possibly, i did a `systemctl reboot` on it | 18:11 |
| fungi | i'll check the server console, and if it's not in progress i'll hard reboot it | 18:11 |
| fungi | A start job is running for OpenAFS client (2min 36s / 3min 21s) | 18:14 |
| *** ykarel|away <ykarel|away!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has quit IRC (Ping timeout: 480 seconds) | 18:14 | |
| fungi | it finally booted | 18:16 |
| clarkb | fungi: looks like the reboot ya | 18:16 |
| *** dviroel <dviroel!uid349012@id-349012.stonehaven.irccloud.com> has quit IRC (Quit: Connection closed for inactivity) | 18:16 | |
| clarkb | zuul configs appear to have loaded I'm restoring queues now | 18:16 |
| fungi | looks like afs didn't completely sync up | 18:16 |
| fungi | https://mirror.regionone.osuosl.opendev.org/ just shows a robots.txt file | 18:16 |
| clarkb | /afs is empty too | 18:16 |
| fungi | ls: cannot access '/afs/openstack.org': No such file or directory | 18:17 |
| fungi | yup | 18:17 |
| fungi | Starting AFS cache scan... Unable to handle kernel paging request at virtual address ffff800016723e40 [...] Internal error: Oops: 96000007 [#1] SMP | 18:18 |
| fungi | i think it got unhappy | 18:18 |
| fungi | i'll give it a second reboot | 18:18 |
| clarkb | ok | 18:18 |
| clarkb | fungi: seems like it is being slow again? I suspect either something on shutdown trying to stop afs units or on startup trying to start them. Then when it gives up things complete and reboot finishes | 18:22 |
| fungi | it's more like it's timing out trying to communicate with the afs servers | 18:22 |
| clarkb | ah | 18:23 |
| fungi | i wonder if there's some sort of udp communications issue there | 18:23 |
| clarkb | reenqueing has completed | 18:25 |
| fungi | looks like it eventually rebooted and is timing out starting afsd again | 18:28 |
| clarkb | it isn't letting me ssh in yet either. I get the pam nologin message | 18:28 |
| fungi | yeah, it won't until it gets past this | 18:28 |
| clarkb | in the oops trace is 'afs_InitCacheFile' | 18:30 |
| clarkb | which implies that maybe something is still up with the cache? | 18:30 |
| fungi | i can try clearing it again, sure | 18:30 |
| clarkb | it certainly seems to have populated | 18:30 |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: Add role to enable FIPS on a node https://review.opendev.org/c/zuul/zuul-jobs/+/788778 | 18:31 |
| fungi | i'm clearing it again and will check more closely once it's done | 18:31 |
| clarkb | fungi: k | 18:31 |
| fungi | ni retrospect i should probably make sure afsd is completely stopped too | 18:32 |
| fungi | in retrospect | 18:32 |
| * fungi is apparently one of the knights who say ni | 18:32 | |
| clarkb | fungi: theory: we mount /var/cache/openafs as another device, I wonder if openafs starting and mounting normal devices race each other and we possible mount over the cache while openafs is operating on it? | 18:33 |
| fungi | ni-wom! | 18:33 |
| fungi | i'll check underneath it, great suggestion | 18:33 |
| clarkb | that could explain errors writing to there too if all of a sudden dirs aren't present or something like that | 18:33 |
| fungi | i can't seem to kill afsd | 18:35 |
| clarkb | I seem to recall this from before and it was cache related then too. | 18:35 |
| fungi | i may need to disable the openafs-client.service unit temporarily and reboot | 18:36 |
| clarkb | fungi: I want to say you can disable it in systemd then reboot | 18:36 |
| clarkb | ya | 18:36 |
| fungi | done and rebooting again | 18:36 |
| fungi | it's still trying to kill afsd to restart | 18:40 |
| clarkb | ya I think that may be why reboots have been slow previously | 18:41 |
| clarkb | since systemd wants to stop all the services as part of that | 18:41 |
| *** david-lyle <david-lyle!~dklyle@134.134.139.72> has joined #opendev | 18:44 | |
| *** dklyle <dklyle!~dklyle@134.134.139.72> has quit IRC (Remote host closed the connection) | 18:44 | |
| fungi | it did eventually reboot | 18:45 |
| fungi | nothing in /var/cache/openafs on its rootfs after unmounting the volume there | 18:46 |
| clarkb | I guess mount the normal cache back, clean it up, then try starting afs manually? | 18:47 |
| fungi | yeah, was just checking that i could ping all our afs servers from it | 18:47 |
| clarkb | oh good idea | 18:48 |
| fungi | cache volume mounted again and starting openafs-client now | 18:48 |
| clarkb | and if this doesn't work maybe we force a dkms rebuild next? | 18:49 |
| fungi | this is arm64 right? | 18:49 |
| clarkb | yes | 18:49 |
| clarkb | it oopsed again | 18:50 |
| clarkb | fwiw linaro is also arm64 | 18:50 |
| clarkb | but possibly different hardware | 18:50 |
| fungi | i'll try package upgrades too just to be sure it's not missing a newer rev | 18:50 |
| clarkb | the trace looks the same as before. its failing in a path to init cache file | 18:51 |
| clarkb | the cache device has plenty of free disk so not a catastrophic handling of no more disk | 18:51 |
| fungi | disabled openafs-client.service and am rebooting again since i couldn't kill afsd and am worried that any openafs package updates might fail postinst scripts if it's stuck | 18:53 |
| clarkb | ++ | 18:53 |
| fungi | and if it's already latest, i'll force a reinstall so dmks rebuild will take place | 18:54 |
| fungi | and will clear the cache volume yet again | 18:54 |
| fungi | i have seen package upgrades in the past timeout/fail dkms builds leaving incomplete or otherwise broken lkms which tehn act weird on the next reboot, so maybe it's that | 18:55 |
| clarkb | fingers crossed | 18:56 |
| fungi | testing to see if it reboots faster when it doesn't have to wait for afsd to not stop | 19:02 |
| fungi | the 10-second grub menu timeout was the longest part of that reboot ;) | 19:03 |
| fungi | so freshly rebooted with afsd not running at all, and `sudo ls -l /var/cache/openafs` took almost 30 seconds | 19:04 |
| fungi | i wonder if there's something not quite right with that volume | 19:04 |
| clarkb | that could also cause problems with cache init | 19:05 |
| fungi | yeah, i'm waiting for it to finish deleting contents again | 19:06 |
| clarkb | fungi: anything I can help with or should I go eat a sandwich really quickly? | 19:10 |
| fungi | go eat, i'm going to reformat the logvol for it | 19:11 |
| clarkb | k | 19:11 |
| fungi | though i have a feeling it's something like terrible iscsi throughput | 19:11 |
| fungi | yeah, slow for sure, even the mkfs.ext4 on that lv is taking a while | 19:16 |
| fungi | and rebooting again for good measure | 19:16 |
| fungi | okay, starting openafs-client again | 19:22 |
| fungi | still not starting | 19:24 |
| fungi | i'll move on to trying the reinstall and dkms rebuild | 19:24 |
| fungi | once it's done oopsing again anyway | 19:25 |
| fungi | yeah, /var/cache/openafs is still megaslow even after reformatting | 19:34 |
| fungi | forcing reinstall of openafs-modules-dkms now | 19:39 |
| clarkb | fungi: do we think the dkms rebuild will help if we suspect a slow volume? maybe we need to try provision a new volume and swap them around and see if that is happier? | 19:40 |
| fungi | i'm doing the dkms rebuild on the chance that the slow volume is unrelated to the afsd startup problem | 19:40 |
| clarkb | ah | 19:41 |
| fungi | okay, that's done, server's rebooted, new openafs lkm is installed, manually starting openafs-client again | 19:54 |
| fungi | it's... taking a while. may still be just as broken as before | 19:55 |
| fungi | yep | 19:55 |
| fungi | kernel:[ 178.804188] Internal error: Oops: 96000007 [#1] SMP | 19:56 |
| clarkb | could be the cache volume then being slow? | 19:56 |
| clarkb | maybe we try replacing it? | 19:56 |
| fungi | probably | 19:56 |
| clarkb | otherwise we're probably at potential bug in openafs on this particular arm hardware that linaro doesn't hit for some reason | 19:58 |
| fungi | removing logical volumes apache and openafs (100gb each) and vg main, along with the pv backing it | 20:13 |
| fungi | mirror01.regionone.osuosl.opendev.org/main detached and deleted | 20:16 |
| fungi | i've added a new cinder volume carved up into 2 lvm logical volumes of the same names as before and reformatted, rebooted to confirm they're automatically mounting and showing the correct available space, trying to start openafs-client again now | 20:22 |
| clarkb | fingers extra crossed | 20:24 |
| fungi | taking a while, i suspect it's no better than before | 20:24 |
| clarkb | :( | 20:24 |
| fungi | yeah, i think it must still be hosed | 20:25 |
| fungi | kernel:[ 208.637324] Internal error: Oops: 96000047 [#1] SMP | 20:25 |
| clarkb | and still in the init cache function? | 20:27 |
| clarkb | its part of the trace | 20:27 |
| *** dviroel <dviroel!uid349012@id-349012.stonehaven.irccloud.com> has joined #opendev | 20:28 | |
| fungi | afs_GetDownDSlot.constprop.0+0xa0/0x1b0 [openafs] | 20:28 |
| fungi | though a couple calls up it's coming from afs_InitCacheFile+0xb0/0x628 [openafs] | 20:28 |
| clarkb | ya that is what it looked like before | 20:28 |
| fungi | i need to switch to dinner prep | 20:32 |
| clarkb | I can get up a change to disable that ergion shortly | 20:32 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Disable the osuosl arm64 cloud https://review.opendev.org/c/openstack/project-config/+/796062 | 20:43 |
| clarkb | fungi: ^ maybe after dinner you can approve that one? otherwise I'll try to remember to approve it in a bit | 20:43 |
| fungi | done, i had a moment while waiting for the skillet to heat up | 20:54 |
| *** donnyd <donnyd!sid368272@id-368272.tooting.irccloud.com> has joined #opendev | 20:54 | |
| *** donnyd_ <donnyd_!~oftc-webi@static-108-44-198-34.clppva.fios.verizon.net> has joined #opendev | 20:55 | |
| opendevreview | Merged openstack/project-config master: Disable the osuosl arm64 cloud https://review.opendev.org/c/openstack/project-config/+/796062 | 21:03 |
| *** donnyd <donnyd!sid368272@id-368272.tooting.irccloud.com> has quit IRC () | 21:06 | |
| *** donnyd <donnyd!sid368272@id-368272.tooting.irccloud.com> has joined #opendev | 21:06 | |
| *** donnyd_ <donnyd_!~oftc-webi@static-108-44-198-34.clppva.fios.verizon.net> has quit IRC (Quit: Page closed) | 21:08 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix some hostnames in afs docs https://review.opendev.org/c/opendev/system-config/+/796064 | 21:21 |
| *** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has quit IRC () | 22:13 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use tmpfiles.d to create /var/run/reprepro https://review.opendev.org/c/opendev/system-config/+/796093 | 22:38 |
| clarkb | I learned a thing that systemd can do today ^ | 22:38 |
| clarkb | I think we probably want ot monitor that just to ensure that a reboot does what we want and no unexpected cleanup happens | 22:41 |
| mordred | clarkb: that doesn't suck | 22:59 |
| clarkb | fungi: ianw: it turns out that ianw has run into this openafs oops before https://www.mail-archive.com/openafs-info@openafs.org/msg41186.html | 23:39 |
| clarkb | seems that 1.8.7 may fix it? maybe we build new package next week? | 23:39 |
| clarkb | or maybe ianw remembers how it was addressed the last time around | 23:40 |
| clarkb | reading the patch that is expected to fix it it does seem like a race between the kernel module setting things up and afsd starting | 23:41 |
| clarkb | I wonder if we can disable afsd, reboot, then force the kernel module to initialize that /proc entry somehow then start afsd | 23:41 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!