| *** betherly has joined #openstack-infra | 00:10 | |
| *** weifan has quit IRC | 00:11 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** weifan has joined #openstack-infra | 00:12 | |
| *** betherly has quit IRC | 00:15 | |
| *** slaweq has quit IRC | 00:15 | |
| *** weifan has quit IRC | 00:16 | |
| donnyd | clarkb: seems like I am getting an above average amount of node errors. Is there a way to look to see what it is? I don't have any errors I can see in the logs ( those I am watching of course) | 00:18 |
|---|---|---|
| clarkb | someone can get the nodepool logd | 00:19 |
| clarkb | I cant asIm transitting to dinner | 00:20 |
| clarkb | ianw may be able to help? | 00:20 |
| *** mattw4 has quit IRC | 00:31 | |
| ianw | umm just a tick but yes | 00:39 |
| *** ijw has quit IRC | 00:42 | |
| *** ramishra has joined #openstack-infra | 00:43 | |
| *** gyee has quit IRC | 00:48 | |
| *** rfolco_l8r is now known as rfolco | 00:49 | |
| openstackgerrit | Qiu Fossen proposed openstack/pbr master: Add Python 3 Train unit tests https://review.opendev.org/670752 | 00:53 |
| openstackgerrit | Qiu Fossen proposed openstack/pbr master: Add Python 3 Train unit tests https://review.opendev.org/670752 | 00:56 |
| ianw | clarkb / donnyd: i'm not really seeing anything on nl02 | 00:57 |
| ianw | packethost-us-west-1 errors, but nothing for fortnebula | 00:57 |
| *** slaweq has joined #openstack-infra | 00:59 | |
| donnyd | that is interesting | 01:00 |
| donnyd | thanks for checking ianw | 01:00 |
| donnyd | When I look here I see quite a few http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 | 01:01 |
| *** icarusfactor has quit IRC | 01:01 | |
| donnyd | I will keep an eye on it | 01:02 |
| *** icarusfactor has joined #openstack-infra | 01:02 | |
| *** betherly has joined #openstack-infra | 01:02 | |
| *** slaweq has quit IRC | 01:04 | |
| ianw | hrm, 22:50 or so | 01:04 |
| ianw | log file i'm looking at starts at 2019-07-16 23:57:58,467 | 01:04 |
| *** betherly has quit IRC | 01:07 | |
| ianw | donnyd: there was some issues around that time; perhaps the timestamps here help? http://paste.openstack.org/show/754467/ | 01:10 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** ruffian_sheep has joined #openstack-infra | 01:14 | |
| *** imacdonn has quit IRC | 01:16 | |
| *** slaweq has quit IRC | 01:16 | |
| *** imacdonn has joined #openstack-infra | 01:16 | |
| *** bobh has joined #openstack-infra | 01:18 | |
| donnyd | Thank you ianw | 01:21 |
| *** dychen has joined #openstack-infra | 01:30 | |
| *** dchen has quit IRC | 01:33 | |
| *** diablo_rojo has quit IRC | 01:34 | |
| *** betherly has joined #openstack-infra | 01:43 | |
| *** betherly has quit IRC | 01:49 | |
| *** apetrich has quit IRC | 01:58 | |
| *** dingyichen has joined #openstack-infra | 02:01 | |
| *** rosmaita has left #openstack-infra | 02:02 | |
| *** dychen has quit IRC | 02:04 | |
| *** aaronsheffield has joined #openstack-infra | 02:16 | |
| *** bobh has quit IRC | 02:24 | |
| *** ruffian_sheep has quit IRC | 02:26 | |
| *** ruffian_sheep has joined #openstack-infra | 02:32 | |
| *** ricolin has joined #openstack-infra | 02:33 | |
| *** icarusfactor has quit IRC | 02:40 | |
| *** icarusfactor has joined #openstack-infra | 02:40 | |
| *** betherly has joined #openstack-infra | 02:46 | |
| *** jamesmcarthur has joined #openstack-infra | 02:48 | |
| *** jamesmcarthur has quit IRC | 02:48 | |
| *** jamesmcarthur has joined #openstack-infra | 02:48 | |
| *** betherly has quit IRC | 02:51 | |
| *** michael-beaver has quit IRC | 02:54 | |
| *** icarusfactor has quit IRC | 02:54 | |
| *** icarusfactor has joined #openstack-infra | 02:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:56 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:59 | |
| *** bhavikdbavishi has quit IRC | 03:01 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:01 | |
| *** betherly has joined #openstack-infra | 03:06 | |
| *** igordc has quit IRC | 03:08 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** betherly has quit IRC | 03:12 | |
| *** icarusfactor has quit IRC | 03:12 | |
| *** icarusfactor has joined #openstack-infra | 03:12 | |
| *** ykarel has joined #openstack-infra | 03:13 | |
| *** icarusfactor has quit IRC | 03:14 | |
| *** icarusfactor has joined #openstack-infra | 03:14 | |
| *** slaweq has quit IRC | 03:16 | |
| *** jbadiapa has quit IRC | 03:21 | |
| *** jbadiapa has joined #openstack-infra | 03:21 | |
| *** icarusfactor has quit IRC | 03:24 | |
| *** factor has joined #openstack-infra | 03:25 | |
| *** jamesmcarthur has quit IRC | 03:26 | |
| *** yamamoto has quit IRC | 03:26 | |
| *** jamesmcarthur has joined #openstack-infra | 03:28 | |
| *** yamamoto has joined #openstack-infra | 03:28 | |
| *** factor has quit IRC | 03:29 | |
| *** factor has joined #openstack-infra | 03:29 | |
| *** pcrews has quit IRC | 03:31 | |
| *** jamesmcarthur has quit IRC | 03:31 | |
| *** psachin has joined #openstack-infra | 03:32 | |
| *** jamesmcarthur has joined #openstack-infra | 03:32 | |
| *** yamamoto has quit IRC | 03:33 | |
| *** betherly has joined #openstack-infra | 03:38 | |
| *** betherly has quit IRC | 03:42 | |
| *** rh-jelabarre has quit IRC | 03:58 | |
| *** udesale has joined #openstack-infra | 04:03 | |
| *** ruffian_sheep has quit IRC | 04:09 | |
| *** betherly has joined #openstack-infra | 04:09 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** yamamoto has joined #openstack-infra | 04:12 | |
| *** ykarel has quit IRC | 04:13 | |
| *** betherly has quit IRC | 04:13 | |
| *** slaweq has quit IRC | 04:16 | |
| *** yamamoto has quit IRC | 04:21 | |
| *** igordc has joined #openstack-infra | 04:22 | |
| *** betherly has joined #openstack-infra | 04:29 | |
| *** bhavikdbavishi has quit IRC | 04:31 | |
| *** jamesmcarthur has quit IRC | 04:33 | |
| *** betherly has quit IRC | 04:34 | |
| *** ykarel has joined #openstack-infra | 04:36 | |
| *** ruffian_sheep has joined #openstack-infra | 04:38 | |
| *** yamamoto has joined #openstack-infra | 04:41 | |
| *** betherly has joined #openstack-infra | 04:50 | |
| *** kopecmartin|off is now known as kopecmartin | 04:54 | |
| *** betherly has quit IRC | 04:54 | |
| *** aaronsheffield has quit IRC | 05:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:17 | |
| *** yamamoto has quit IRC | 05:17 | |
| *** yamamoto has joined #openstack-infra | 05:28 | |
| *** udesale has quit IRC | 05:35 | |
| *** udesale has joined #openstack-infra | 05:35 | |
| *** udesale has quit IRC | 05:36 | |
| *** udesale has joined #openstack-infra | 05:36 | |
| *** ykarel_ has joined #openstack-infra | 05:36 | |
| *** ykarel has quit IRC | 05:38 | |
| *** betherly has joined #openstack-infra | 05:41 | |
| *** betherly has quit IRC | 05:46 | |
| *** whoami-rajat has joined #openstack-infra | 06:06 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** igordc has quit IRC | 06:11 | |
| *** betherly has joined #openstack-infra | 06:12 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** igordc has joined #openstack-infra | 06:15 | |
| *** pgaxatte has joined #openstack-infra | 06:16 | |
| *** slaweq has quit IRC | 06:16 | |
| *** betherly has quit IRC | 06:17 | |
| *** iurygregory has joined #openstack-infra | 06:18 | |
| *** liuyulong has quit IRC | 06:19 | |
| *** raukadah is now known as chandankumar | 06:19 | |
| *** pcaruana has joined #openstack-infra | 06:20 | |
| *** ruffian_sheep has quit IRC | 06:21 | |
| *** udesale has quit IRC | 06:22 | |
| *** udesale has joined #openstack-infra | 06:22 | |
| *** joeguo has quit IRC | 06:24 | |
| *** udesale has quit IRC | 06:24 | |
| *** udesale has joined #openstack-infra | 06:25 | |
| *** slaweq has joined #openstack-infra | 06:28 | |
| *** rakhmerov has joined #openstack-infra | 06:29 | |
| *** slaweq has quit IRC | 06:33 | |
| *** igordc has quit IRC | 06:33 | |
| *** dingyichen has quit IRC | 06:36 | |
| *** dchen has joined #openstack-infra | 06:37 | |
| *** apetrich has joined #openstack-infra | 06:40 | |
| *** betherly has joined #openstack-infra | 06:43 | |
| *** ruffian_sheep has joined #openstack-infra | 06:45 | |
| *** betherly has quit IRC | 06:48 | |
| *** jtomasek has joined #openstack-infra | 06:49 | |
| *** yamamoto has joined #openstack-infra | 06:51 | |
| *** aluria has joined #openstack-infra | 07:00 | |
| *** slaweq has joined #openstack-infra | 07:01 | |
| *** yamamoto has quit IRC | 07:02 | |
| *** betherly has joined #openstack-infra | 07:04 | |
| *** rpittau|afk is now known as rpittau | 07:08 | |
| *** betherly has quit IRC | 07:09 | |
| *** yamamoto has joined #openstack-infra | 07:13 | |
| *** ralonsoh has joined #openstack-infra | 07:15 | |
| *** gfidente has joined #openstack-infra | 07:18 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 07:18 | |
| *** ginopc has joined #openstack-infra | 07:21 | |
| *** xek has joined #openstack-infra | 07:26 | |
| *** tosky has joined #openstack-infra | 07:26 | |
| openstackgerrit | Igor D.C. proposed ttygroup/gertty master: Make browser open on Windows when running in cygwin https://review.opendev.org/671202 | 07:29 |
| openstackgerrit | Igor D.C. proposed ttygroup/gertty master: Make browser open on Windows when running in cygwin https://review.opendev.org/671202 | 07:32 |
| openstackgerrit | Igor D.C. proposed ttygroup/gertty master: Make browser open on Windows when running in cygwin https://review.opendev.org/671202 | 07:33 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Update examples in MQTT reporter docu https://review.opendev.org/671203 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Record and report time for trigger events https://review.opendev.org/671204 | 07:36 |
| *** ykarel_ is now known as ykarel|meeting | 07:42 | |
| *** yamamoto has quit IRC | 07:48 | |
| *** kjackal has joined #openstack-infra | 07:50 | |
| *** ykarel has joined #openstack-infra | 07:55 | |
| *** rcernin has quit IRC | 07:56 | |
| *** udesale has quit IRC | 07:56 | |
| *** odicha has joined #openstack-infra | 07:56 | |
| *** ykarel|meeting has quit IRC | 07:56 | |
| *** ykarel is now known as ykarel|lunch | 07:57 | |
| *** odicha has quit IRC | 07:58 | |
| *** odicha has joined #openstack-infra | 07:58 | |
| *** lucasagomes has joined #openstack-infra | 07:59 | |
| *** zhurong has quit IRC | 08:00 | |
| *** dchen has quit IRC | 08:04 | |
| *** yamamoto has joined #openstack-infra | 08:09 | |
| *** pkopec has joined #openstack-infra | 08:09 | |
| *** yolanda has quit IRC | 08:12 | |
| *** rcernin has joined #openstack-infra | 08:12 | |
| *** yolanda has joined #openstack-infra | 08:12 | |
| *** yolanda has quit IRC | 08:12 | |
| openstackgerrit | Merged zuul/zuul master: Unify gearman worker handling https://review.opendev.org/664948 | 08:17 |
| *** joeguo has joined #openstack-infra | 08:18 | |
| *** betherly has joined #openstack-infra | 08:19 | |
| *** dtantsur|afk is now known as dtantsur | 08:23 | |
| *** betherly has quit IRC | 08:24 | |
| *** Fidde has joined #openstack-infra | 08:30 | |
| *** zhurong has joined #openstack-infra | 08:31 | |
| *** ykarel|lunch is now known as ykarel | 08:39 | |
| *** betherly has joined #openstack-infra | 08:40 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Move fingergw config to fingergw https://review.opendev.org/664949 | 08:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Route streams to different zones via finger gateway https://review.opendev.org/664965 | 08:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support ssl encrypted fingergw https://review.opendev.org/664950 | 08:40 |
| *** electrofelix has joined #openstack-infra | 08:44 | |
| *** betherly has quit IRC | 08:44 | |
| *** udesale has joined #openstack-infra | 08:46 | |
| *** e0ne has joined #openstack-infra | 08:47 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 08:53 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 08:53 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 08:53 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 08:53 |
| *** ociuhandu has joined #openstack-infra | 08:57 | |
| *** ociuhandu has quit IRC | 08:58 | |
| *** derekh has joined #openstack-infra | 08:58 | |
| *** joeguo has quit IRC | 09:01 | |
| *** ociuhandu has joined #openstack-infra | 09:04 | |
| *** tkajinam has quit IRC | 09:09 | |
| *** ociuhandu has quit IRC | 09:10 | |
| *** ginopc has quit IRC | 09:20 | |
| *** dikonoor has joined #openstack-infra | 09:25 | |
| *** joeguo has joined #openstack-infra | 09:49 | |
| *** e0ne has quit IRC | 10:00 | |
| *** kjackal has quit IRC | 10:01 | |
| *** ruffian_sheep has quit IRC | 10:03 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:03 | |
| *** sshnaidm is now known as sshnaidm|rover | 10:03 | |
| *** ociuhandu has joined #openstack-infra | 10:05 | |
| jonher | I got a internal server error on https://opendev.org/x/networking-fortinet/ just a couple of minutes ago, is there still a problem with the gitea backends? | 10:06 |
| *** kjackal has joined #openstack-infra | 10:11 | |
| *** ociuhandu has quit IRC | 10:13 | |
| *** kjackal has quit IRC | 10:24 | |
| *** dikonoor has quit IRC | 10:38 | |
| *** udesale has quit IRC | 10:38 | |
| *** owalsh has quit IRC | 10:42 | |
| *** slaweq has quit IRC | 10:43 | |
| *** slaweq has joined #openstack-infra | 10:43 | |
| *** priteau has joined #openstack-infra | 10:45 | |
| *** ociuhandu has joined #openstack-infra | 10:46 | |
| *** betherly has joined #openstack-infra | 10:50 | |
| *** xek has quit IRC | 10:55 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Use a requests session to simplify auth'd calls https://review.opendev.org/670511 | 10:55 |
| *** ociuhandu has quit IRC | 10:57 | |
| *** xek has joined #openstack-infra | 10:58 | |
| *** kjackal has joined #openstack-infra | 10:59 | |
| *** kjackal has quit IRC | 11:06 | |
| sshnaidm|rover | AJaeger, when I use "debug" in pipeline, where can I see output of that? https://review.opendev.org/#/c/671055/3/zuul.d/layout.yaml | 11:09 |
| sshnaidm|rover | I don't see jobs running at all | 11:10 |
| AJaeger | sshnaidm|rover: wait until Zuul reports back, it's in gerrit and the mail you recieve | 11:13 |
| *** Lucas_Gray has joined #openstack-infra | 11:14 | |
| sshnaidm|rover | AJaeger, 5 hours passed | 11:15 |
| *** gtarnaras has joined #openstack-infra | 11:17 | |
| *** owalsh has joined #openstack-infra | 11:18 | |
| *** ykarel is now known as ykarel|afk | 11:27 | |
| *** tesseract has joined #openstack-infra | 11:28 | |
| *** rcernin has quit IRC | 11:30 | |
| *** rosmaita has joined #openstack-infra | 11:31 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Document the info endpoints https://review.opendev.org/671243 | 11:37 |
| *** panda is now known as panda|lunch | 11:44 | |
| *** gtarnaras has quit IRC | 11:47 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support ssl encrypted fingergw https://review.opendev.org/664950 | 11:49 |
| mordred | knikolla: morning! I have a question which is going to sound like something that I should know, but let's just assume I'm dumb some times ... | 11:49 |
| *** rh-jelabarre has joined #openstack-infra | 11:49 | |
| mordred | knikolla: how do I make an application credential? I don't see anything in the dashboard - do I have to do something with osc and oidc with my rh sso? | 11:50 |
| mordred | knikolla: also, don't tell cmurphy I asked this ... ;) | 11:50 |
| openstackgerrit | Merged zuul/zuul master: Update examples in MQTT reporter docu https://review.opendev.org/671203 | 11:51 |
| openstackgerrit | Benedikt Löffler proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 11:51 |
| openstackgerrit | Benedikt Löffler proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 11:51 |
| *** gtarnaras has joined #openstack-infra | 11:51 | |
| lennyb | AJaeger,fungi, regarding #link https://docs.openstack.org/infra/manual/creators.html#mirroring-projects-to-git-mirrors .. how can this work if this job is in my CI, and zuul who actually merges the code is in infra environment? Or I should propose a patch for infra, but in that case I will expose my github credentials | 11:55 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 12:03 |
| *** ociuhandu has joined #openstack-infra | 12:04 | |
| *** kjackal has joined #openstack-infra | 12:06 | |
| fungi | lennyb: the patch would include an ssh private key encrypted as a "zuul secret" so that only zuul can decrypt and make use of it: https://docs.openstack.org/infra/manual/zuulv3.html#secret-variables | 12:06 |
| lennyb | fungi, thanks. | 12:07 |
| fungi | lennyb: also i strongly recommend it not be an ssh key for your personal account, but rather that you create a separate account specifically for this purpose, authorized to push into the mirror repository | 12:08 |
| lennyb | fungi, our project is #link https://github.com/openstack/networking-mlnx , it's under openstack organization. Does infra team some kind of automatic procedure for this since it's not 'really' external mirror ? | 12:11 |
| fungi | lennyb: yes, first you would need to decide what organization on github you want it transferred into, and then work with us to perform the transfer in github | 12:13 |
| *** _erlon_ has joined #openstack-infra | 12:13 | |
| *** ekultails has joined #openstack-infra | 12:13 | |
| fungi | we won't be mirroring it into https://github.com/openstack/networking-mlnx any longer, but rather https://github.com/your_new_org/networking-mlnx | 12:13 |
| fungi | and https://github.com/openstack/networking-mlnx will become a redirect to https://github.com/your_new_org/networking-mlnx | 12:14 |
| *** ociuhandu has quit IRC | 12:14 | |
| lennyb | fungi, can it be possible to redirect https://github.com/openstack/networking-mlnx back to https://opendev.org/x/networking-mlnx ? | 12:18 |
| fungi | nope | 12:18 |
| fungi | github only allows you to redirect to other places in github (using repo renames and transfers) | 12:19 |
| *** roman_g has quit IRC | 12:20 | |
| lennyb | so, if we wont create "your_new_org" github it just will be deleted and we will stay with opendev.org repo only.. | 12:21 |
| *** apetrich has quit IRC | 12:21 | |
| fungi | lennyb: oh, if you don't want a mirror on github that's even easier. we can simply delete the stale copy at https://github.com/openstack/networking-mlnx | 12:21 |
| *** roman_g has joined #openstack-infra | 12:22 | |
| *** ykarel|afk is now known as ykarel | 12:25 | |
| lennyb | fungi, I will update you. Let me check. BTW, What about other projects? I saw that at least few of them are not mirroring data back to github. | 12:26 |
| *** sshnaidm_ has joined #openstack-infra | 12:30 | |
| *** sshnaidm|rover has quit IRC | 12:30 | |
| *** goldyfruit has quit IRC | 12:30 | |
| *** sshnaidm_ is now known as sshnaidm|rover | 12:30 | |
| *** apetrich has joined #openstack-infra | 12:31 | |
| *** udesale has joined #openstack-infra | 12:33 | |
| *** yamamoto has quit IRC | 12:37 | |
| *** panda|lunch is now known as panda | 12:37 | |
| * mordred is working on a script to handle the things in openstack-infra ... | 12:39 | |
| *** yamamoto has joined #openstack-infra | 12:41 | |
| *** ijw has joined #openstack-infra | 12:44 | |
| *** yamamoto has quit IRC | 12:46 | |
| fungi | lennyb: jroll has a plan to mark them all "archived" in github | 12:48 |
| fungi | in case other folks also decide they want to transfer their repos to new github orgs | 12:48 |
| *** ijw has quit IRC | 12:49 | |
| *** Lucas_Gray has quit IRC | 12:59 | |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| *** Lucas_Gray has joined #openstack-infra | 13:04 | |
| *** hamzy has quit IRC | 13:05 | |
| *** aaronsheffield has joined #openstack-infra | 13:05 | |
| *** yamamoto has quit IRC | 13:08 | |
| AJaeger | sshnaidm|rover: what is the change you tried it on? | 13:10 |
| sshnaidm|rover | AJaeger, on this: https://review.opendev.org/#/c/671055/ now it's running something with "recheck" | 13:15 |
| *** jamesdenton has quit IRC | 13:16 | |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Add puppet-crane noop jobs https://review.opendev.org/671263 | 13:17 |
| AJaeger | sshnaidm|rover: then let's wait until Zuul has finished - only then you get the information | 13:17 |
| *** jamesdenton has joined #openstack-infra | 13:17 | |
| sshnaidm|rover | AJaeger, ack | 13:18 |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Add puppet-crane noop jobs https://review.opendev.org/671263 | 13:19 |
| *** artom has quit IRC | 13:19 | |
| knikolla | mordred: it's actually a valid question, cause the version of horizon we have (i think queens) doesn't have that feature yet. | 13:23 |
| knikolla | mordred: use this to log in with OIDC https://osticket.massopen.cloud/scp/faq.php?id=16 | 13:23 |
| knikolla | and create and app cred from the API | 13:23 |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Remove puppet-crane https://review.opendev.org/671268 | 13:23 |
| *** liuyulong has joined #openstack-infra | 13:24 | |
| *** ykarel is now known as ykarel|afk | 13:25 | |
| *** liuyulong has quit IRC | 13:26 | |
| mordred | knikolla: sweet, thanks! | 13:26 |
| mordred | knikolla: I don't seem to have a login to that? | 13:27 |
| knikolla | mordred: https://osticket.massopen.cloud/kb/faq.php?id=16 | 13:27 |
| knikolla | sorry, sent you the admin panel link | 13:27 |
| mordred | knikolla: yay! that one works | 13:27 |
| *** ykarel|afk has quit IRC | 13:32 | |
| *** jamesmcarthur has joined #openstack-infra | 13:32 | |
| *** bhavikdbavishi has quit IRC | 13:33 | |
| *** rajinir has joined #openstack-infra | 13:34 | |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Add puppet-crane noop jobs https://review.opendev.org/671263 | 13:34 |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Remove puppet-crane https://review.opendev.org/671268 | 13:34 |
| *** hamzy has joined #openstack-infra | 13:36 | |
| *** psachin has quit IRC | 13:37 | |
| *** tdasilva has joined #openstack-infra | 13:37 | |
| *** goldyfruit has joined #openstack-infra | 13:38 | |
| *** Lucas_Gray has quit IRC | 13:40 | |
| zbr | i need some insights about how it would be able to test changes made to roles hosted inside a trusted repo with zuul. | 13:40 |
| *** yamamoto has joined #openstack-infra | 13:42 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** Lucas_Gray has joined #openstack-infra | 13:43 | |
| *** udesale has quit IRC | 13:43 | |
| *** udesale has joined #openstack-infra | 13:43 | |
| *** gtarnaras has quit IRC | 13:45 | |
| *** gtarnaras has joined #openstack-infra | 13:51 | |
| *** jamesmcarthur has quit IRC | 13:54 | |
| *** jamesmcarthur has joined #openstack-infra | 13:56 | |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Remove puppet-crane https://review.opendev.org/671268 | 13:57 |
| *** jamesmcarthur has quit IRC | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:01 | |
| openstackgerrit | Corey Bryant proposed openstack/pbr master: Add Python 3 Train unit tests https://review.opendev.org/669467 | 14:03 |
| donnyd | so I had a hypervisor burn up its cpu's last night, so I am backing off the oversubscription of cores a bit. Hopefully this also solves the timeout issues I have been hitting on FN. Shouldn't really effect anything, just an FYI | 14:04 |
| *** e0ne has joined #openstack-infra | 14:05 | |
| fungi | burn up literally? the magic smoke escaped? | 14:08 |
| *** altlogbot_3 has quit IRC | 14:09 | |
| *** irclogbot_3 has quit IRC | 14:17 | |
| *** ricolin_ has joined #openstack-infra | 14:17 | |
| *** altlogbot_0 has joined #openstack-infra | 14:17 | |
| *** ykarel has joined #openstack-infra | 14:19 | |
| donnyd | yea | 14:19 |
| donnyd | like server won't boot, throws CPU 4 FAILED at BIOS | 14:19 |
| *** ricolin has quit IRC | 14:19 | |
| donnyd | LOL, never had that issue before, but out with the weak | 14:20 |
| donnyd | I am going to have to turn down FN a little bit, it would seem nodepool is not honoring the quota reduction | 14:20 |
| *** altlogbot_0 has quit IRC | 14:22 | |
| *** yamamoto has joined #openstack-infra | 14:22 | |
| *** udesale has quit IRC | 14:25 | |
| *** altlogbot_0 has joined #openstack-infra | 14:25 | |
| fungi | hrm, it ought to see the quotas reported by the api and avoid exceeding them | 14:26 |
| *** altlogbot_0 has quit IRC | 14:26 | |
| *** jamesmcarthur has quit IRC | 14:26 | |
| fungi | but yeah, http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 shows it thinking it can boot up to 60 nodes there | 14:26 |
| *** altlogbot_0 has joined #openstack-infra | 14:26 | |
| fungi | which is the current max-servers configured in nodepool | 14:27 |
| donnyd | It also may be a cron job that take a bit to sync or something | 14:27 |
| fungi | are you sure you reduced the quota on the correct tenant? | 14:28 |
| donnyd | yea | 14:29 |
| * fungi refamiliarizes himself with how to inspect quotas via osc | 14:29 | |
| *** gtarnaras has quit IRC | 14:29 | |
| donnyd | I will watch the logs for a little while, but its at quota right now and I keep getting Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up. | 14:30 |
| *** ykarel has quit IRC | 14:31 | |
| clarkb | fungi `openstack limits show` or list iirc | 14:31 |
| *** altlogbot_0 has quit IRC | 14:32 | |
| fungi | ahh, i had gotten to `openstack quota list --compute` | 14:32 |
| fungi | but yeah, i guess these are technically limits and not quotas? | 14:32 |
| clarkb | quota list is admin iirc | 14:32 |
| *** yamamoto has quit IRC | 14:33 | |
| clarkb | there is some confusing reason they are different | 14:33 |
| fungi | yep. `openstack limits show --absolute` to be precise | 14:33 |
| *** altlogbot_1 has joined #openstack-infra | 14:33 | |
| donnyd | I also had the quota pretty close to the edge of how things are currently setup | 14:34 |
| fungi | here's how it looks to us: http://paste.openstack.org/show/754503/ | 14:34 |
| corvus | fungi, donnyd: i think the max line in the graph is just based on 'max-servers', but it should still not try to boot more than the quota. iow, the max line being at 60 is normal; all the other lines not being at 60 means something's weird. | 14:35 |
| corvus | (er, all the other lines *being* at 60 means something weird) | 14:35 |
| donnyd | I had it backed off to 59 and the system can currently do 60 | 14:35 |
| fungi | corvus: right, i was going by the red line being basically at 60 | 14:35 |
| donnyd | so i backed it down to 57 and it looks to have stopped | 14:35 |
| fungi | i guess it's 59 but on the graph it's hard to see the difference | 14:35 |
| fungi | (red line being deleting float) | 14:36 |
| fungi | looks like maybe it dropped to 59 around 12:00z | 14:36 |
| corvus | oh, also, nodepool isn't perfect (it can be slightly racy), so +/- 2 wouldn't shock me. | 14:36 |
| fungi | currently the api is reporting 57 so nodepool shouldn't try to boot more there until used drops below 57 | 14:37 |
| fungi | but right, it can have a bit of hysteresis around the setpoint | 14:37 |
| donnyd | I will keep watching the logs here, but i haven't seen it since I dropped it to 57 from 59 | 14:37 |
| *** altlogbot_1 has quit IRC | 14:38 | |
| donnyd | Hopefully now the tempest jobs stop timing out | 14:38 |
| zbr | clarkb: fungi : any chance to look at https://review.opendev.org/#/c/671029/1 ? is not big but saves a lot of copy/paste in other repos. | 14:39 |
| *** altlogbot_2 has joined #openstack-infra | 14:39 | |
| fungi | as for a dead cpu taking the entire server offline, i rather miss the old proliant dl series servers i had at home with cpu redundancy. could lose an entire processor and still keep chugging along (at half capacity of course) | 14:40 |
| fungi | but man those things sure sucked some juice, generated a ton of heat and the psu fans were deafening | 14:40 |
| fungi | and redundant psus meant twice the ear damage | 14:40 |
| *** altlogbot_2 has quit IRC | 14:44 | |
| corvus | fungi, clarkb, mordred: can you review 670920 and children? | 14:44 |
| corvus | that's the gitea stack from yesterday | 14:44 |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Add release jobs for ansible-role-collect-logs https://review.opendev.org/671311 | 14:44 |
| fungi | oh, yep i had already started looking at those before i passed out last night | 14:44 |
| donnyd | I have very little in the way of complaints for my m1000e chassis. There are struggles here and there, but for the most part everything works as it should (except storage, its sucks) | 14:45 |
| fungi | thanks for the reminder! | 14:45 |
| *** bhavikdbavishi has quit IRC | 14:45 | |
| *** altlogbot_1 has joined #openstack-infra | 14:45 | |
| donnyd | anyone curious to see what the load on the hypervisor looks like? | 14:47 |
| *** factor has quit IRC | 14:47 | |
| donnyd | https://usercontent.irccloud-cdn.com/file/a53Ly0oR/image.png | 14:48 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add dstat graph roles https://review.opendev.org/670402 | 14:49 |
| donnyd | seems like my CPU's have more to give, but i already was doing that and it didn't work out well | 14:49 |
| *** jamesmcarthur has joined #openstack-infra | 14:49 | |
| *** altlogbot_1 has quit IRC | 14:50 | |
| *** mattw4 has joined #openstack-infra | 14:51 | |
| *** altlogbot_1 has joined #openstack-infra | 14:51 | |
| fungi | that does look fairly light still, yep | 14:53 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 14:54 |
| *** ykarel has joined #openstack-infra | 14:55 | |
| *** altlogbot_1 has quit IRC | 14:56 | |
| *** altlogbot_1 has joined #openstack-infra | 14:57 | |
| *** factor has joined #openstack-infra | 14:59 | |
| *** gyee has joined #openstack-infra | 14:59 | |
| *** pgaxatte has quit IRC | 15:00 | |
| *** altlogbot_1 has quit IRC | 15:02 | |
| *** igordc has joined #openstack-infra | 15:02 | |
| mordred | corvus: fwiw - re: adding gerrit, I think I'm on the "we don't really need to" boat | 15:03 |
| mordred | corvus: I don't think we gain a ton by making one job that does both at the same time - vs two jobs that do the specific thing | 15:03 |
| *** altlogbot_2 has joined #openstack-infra | 15:03 | |
| *** eernst has joined #openstack-infra | 15:04 | |
| mordred | except to verify that we run gitea before gerrit - and that is super unlikley to accidentally change | 15:04 |
| *** bhavikdbavishi has joined #openstack-infra | 15:05 | |
| corvus | mordred: yeah? it seemed to work and only added 3 minutes, so i could probably go either way ... | 15:06 |
| *** ociuhandu has joined #openstack-infra | 15:06 | |
| corvus | i'll hack together the alternative in a minute | 15:07 |
| *** altlogbot_2 has quit IRC | 15:08 | |
| donnyd | fungi: yea, its still around 50% util | 15:09 |
| donnyd | but also i have no idea what jobs are running and cpus are mapped 1:1.5 | 15:09 |
| donnyd | It was 2.5 before, and that seemed to be too much | 15:09 |
| mordred | corvus: yeah - I could go either way too | 15:09 |
| mordred | corvus: I think that was more "I don't think we need to" - but I'm not *opposed* to doing both in the same job either | 15:10 |
| donnyd | if 1.5 is too low, maybe 2 is the ideal spot.. they should be around 75% util | 15:10 |
| *** priteau has quit IRC | 15:11 | |
| *** gtarnaras has joined #openstack-infra | 15:13 | |
| *** altlogbot_3 has joined #openstack-infra | 15:13 | |
| *** tosky has quit IRC | 15:13 | |
| *** kjackal has quit IRC | 15:16 | |
| *** altlogbot_3 has quit IRC | 15:18 | |
| *** ociuhandu has quit IRC | 15:19 | |
| *** altlogbot_2 has joined #openstack-infra | 15:19 | |
| *** factor has quit IRC | 15:20 | |
| *** factor has joined #openstack-infra | 15:22 | |
| *** jamesmcarthur has quit IRC | 15:24 | |
| *** odicha has quit IRC | 15:24 | |
| *** eernst has quit IRC | 15:26 | |
| *** altlogbot_2 has quit IRC | 15:28 | |
| *** altlogbot_2 has joined #openstack-infra | 15:33 | |
| *** ykarel is now known as ykarel|afk | 15:36 | |
| *** Fidde has quit IRC | 15:36 | |
| *** altlogbot_2 has quit IRC | 15:38 | |
| *** altlogbot_3 has joined #openstack-infra | 15:39 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add add-gpgkey test job https://review.opendev.org/670206 | 15:40 |
| *** altlogbot_3 has quit IRC | 15:44 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add run-gerrit job https://review.opendev.org/671323 | 15:45 |
| corvus | mordred: ^ there's the alt | 15:45 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add dstat graph roles https://review.opendev.org/670402 | 15:45 |
| *** altlogbot_3 has joined #openstack-infra | 15:45 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add gerrit to gitea job https://review.opendev.org/671162 | 15:46 |
| *** ykarel|afk is now known as ykarel|away | 15:47 | |
| *** altlogbot_3 has quit IRC | 15:50 | |
| *** altlogbot_1 has joined #openstack-infra | 15:51 | |
| mordred | corvus: well - an argument for combined is that there's only one copy of bridge in the combined job | 15:55 |
| mordred | corvus: whereas as two jobs it has to allocate a node for bridge twice | 15:55 |
| corvus | mordred: yeah... sometimes we're going to run both jobs, but other times we'd only run one | 15:56 |
| corvus | (if we modify the playbook, we need to run both; if we modify one of the roles, only one) | 15:56 |
| fungi | mordred: does 671000 need to be rebased? gerrit says it has an outdated parent | 15:56 |
| mordred | corvus: yeah | 15:56 |
| mordred | corvus: I have no idea which way I think is "better" | 15:57 |
| *** altlogbot_1 has quit IRC | 15:57 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Silence InsecureRequestWarning and password warning https://review.opendev.org/671000 | 15:57 |
| mordred | fungi: yes! | 15:58 |
| fungi | cool, thanks! | 15:58 |
| *** e0ne has quit IRC | 16:00 | |
| *** altlogbot_0 has joined #openstack-infra | 16:03 | |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| *** altlogbot_0 has quit IRC | 16:08 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add add-launchpad-credentaials test job https://review.opendev.org/670207 | 16:08 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add add-sshkey test job https://review.opendev.org/670208 | 16:09 |
| *** altlogbot_2 has joined #openstack-infra | 16:09 | |
| *** lucasagomes has quit IRC | 16:11 | |
| *** irclogbot_3 has joined #openstack-infra | 16:13 | |
| *** altlogbot_2 has quit IRC | 16:14 | |
| *** altlogbot_0 has joined #openstack-infra | 16:15 | |
| *** dims has quit IRC | 16:15 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 16:15 | |
| *** Lucas_Gray has quit IRC | 16:17 | |
| *** irclogbot_3 has quit IRC | 16:17 | |
| *** ijw has joined #openstack-infra | 16:17 | |
| *** ricolin_ is now known as ricolin | 16:20 | |
| *** icarusfactor has joined #openstack-infra | 16:21 | |
| *** factor has quit IRC | 16:21 | |
| *** icarusfactor has quit IRC | 16:21 | |
| clarkb | ianw has a couple docs changes at https://review.opendev.org/#/c/668833/ and https://review.opendev.org/#/c/669602/1 the second of which is for LE which would be good to get in | 16:22 |
| clarkb | corvus: did we want to get more of the 670920 stack in before approving https://review.opendev.org/#/c/668681/ ? | 16:22 |
| *** gtarnaras has quit IRC | 16:22 | |
| *** irclogbot_1 has joined #openstack-infra | 16:23 | |
| corvus | clarkb: yeah, i'd like to go all the way to 160 at least | 16:25 |
| clarkb | ok I'm catching up on those reviews now (slow start to the day) | 16:26 |
| *** irclogbot_1 has quit IRC | 16:26 | |
| *** betherly has quit IRC | 16:26 | |
| *** irclogbot_2 has joined #openstack-infra | 16:27 | |
| clarkb | corvus: for https://review.opendev.org/#/c/671000/3//COMMIT_MSG what makes the request insecure? is it that we connect to localhost which doesn't match the ssl cert for opendev.org? | 16:27 |
| clarkb | or rather why does it think it is insecure | 16:27 |
| mordred | knikolla: I get "RESP BODY: {"error":"invalid_grant","error_description":"Invalid user credentials"}" when I try that with motaylor@redhat.com and my RH/google SSO creds - is there maybe a different set of settings for rh/google/saml ? | 16:27 |
| openstackgerrit | Merged opendev/system-config master: Use a thread pool to update gitea repos faster https://review.opendev.org/670920 | 16:28 |
| knikolla | mordred: did you set a password? | 16:29 |
| mordred | knikolla: oh - hah! I now see that step | 16:29 |
| mordred | knikolla: how about I actually follow the actual directions? | 16:29 |
| *** irclogbot_2 has quit IRC | 16:30 | |
| corvus | clarkb: we use a self-signed cert in testing | 16:30 |
| clarkb | corvus: ah | 16:30 |
| *** irclogbot_2 has joined #openstack-infra | 16:31 | |
| fungi | and this is easier that explicitly adding it to the trust store | 16:32 |
| fungi | er, easier THAN | 16:32 |
| mordred | knikolla: ok. that all worked - thank you for your help! | 16:32 |
| openstackgerrit | Merged opendev/system-config master: Provide better module return info from gitea create repos https://review.opendev.org/671159 | 16:33 |
| *** irclogbot_2 has quit IRC | 16:34 | |
| *** altlogbot_0 has quit IRC | 16:34 | |
| *** irclogbot_3 has joined #openstack-infra | 16:35 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add add-gpgkey test job https://review.opendev.org/670206 | 16:36 |
| *** altlogbot_2 has joined #openstack-infra | 16:36 | |
| clarkb | corvus: for https://review.opendev.org/#/c/671160/1/playbooks/roles/gitea-git-repos/library/gitea_create_repos.py if there are no updates to be made we degrade to the serial case, but since there are no updates anyway that is still quick? | 16:36 |
| mordred | it should be | 16:37 |
| corvus | clarkb: yep, there's no network traffic, so it takes about the same amount of time (we're GIL constrained either way) | 16:37 |
| *** irclogbot_3 has quit IRC | 16:38 | |
| clarkb | for the test gerrit with gitea or not question I kind of like the idea of testing playbooks as a functional unit (so testing gerrit with gitea makes sense to me) | 16:38 |
| *** irclogbot_2 has joined #openstack-infra | 16:39 | |
| corvus | yeah i think that's going to be the general pattern | 16:40 |
| *** irclogbot_2 has quit IRC | 16:42 | |
| *** irclogbot_2 has joined #openstack-infra | 16:43 | |
| clarkb | I've +2'd https://review.opendev.org/#/c/671162/3 since I think that is my preference | 16:44 |
| clarkb | unless we split out playbooks further then we can split the jobs | 16:44 |
| mordred | I think I was on the fence enough that that argument is enough to convince me (being the general pattern) | 16:44 |
| *** irclogbot_2 has quit IRC | 16:46 | |
| *** irclogbot_0 has joined #openstack-infra | 16:47 | |
| corvus | mordred: the puppet run on fake-review didn't seem to do anything: http://logs.openstack.org/62/671162/3/check/system-config-run-git/e99b0a1/job-output.txt.gz#_2019-07-17_16_25_48_312214 | 16:48 |
| *** irclogbot_0 has quit IRC | 16:50 | |
| *** irclogbot_1 has joined #openstack-infra | 16:51 | |
| *** irclogbot_1 has quit IRC | 16:54 | |
| *** irclogbot_2 has joined #openstack-infra | 16:55 | |
| *** ykarel|away has quit IRC | 16:55 | |
| *** jbadiapa has quit IRC | 16:55 | |
| clarkb | infra-root I'm going to run https://review.opendev.org/#/c/671102/ on gitea06 shortly (mostly a heads up) | 16:55 |
| clarkb | that should add a working swapfile to that server and hopefully make the OOMs go away there | 16:56 |
| mordred | clarkb: fingers crossed | 16:57 |
| *** irclogbot_2 has quit IRC | 16:58 | |
| mordred | knikolla: next issue: | 16:58 |
| mordred | montytaylor@penguin:~$ ./venv/bin/openstack application credential list | 16:59 |
| mordred | public endpoint for identity service in MOC_Kaizen region not found | 16:59 |
| *** irclogbot_0 has joined #openstack-infra | 16:59 | |
| mordred | knikolla: region name should be moc-kzn not MOC_Kaizen | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| knikolla | mordred: correct! will update documentation now as that is out of date. | 17:00 |
| mordred | woot! | 17:00 |
| * mordred is helpful | 17:00 | |
| knikolla | \o/ | 17:01 |
| mordred | knikolla: I can now successfully execute commands as me - I'll make an appcred next | 17:01 |
| openstackgerrit | Merged opendev/system-config master: Parallelize repo creation by org https://review.opendev.org/671160 | 17:01 |
| mordred | knikolla: fwiw - this is the first time I've ever used OIDC with an openstack cloud - so I'm pretty excited it all just works | 17:01 |
| mordred | cmurphy: ^^ | 17:01 |
| *** irclogbot_0 has quit IRC | 17:02 | |
| *** irclogbot_2 has joined #openstack-infra | 17:03 | |
| cmurphy | it is indeed shocking when software works | 17:04 |
| knikolla | hahaha | 17:04 |
| *** irclogbot_2 has quit IRC | 17:06 | |
| *** irclogbot_3 has joined #openstack-infra | 17:07 | |
| *** irclogbot_3 has quit IRC | 17:10 | |
| *** irclogbot_1 has joined #openstack-infra | 17:11 | |
| clarkb | apparently mount -a with a swapfile entry in /etc/fstab isn't enough to get swap enabled and you have to swapon? Maybe I should remove gitea06 from the lb temporarily and reboot it to double check this works on a reboot? | 17:11 |
| clarkb | fungi: ^ is there something I'm missing there? | 17:11 |
| fungi | did you mkswap first? | 17:13 |
| *** irclogbot_1 has quit IRC | 17:14 | |
| fungi | i need to go grab a late lunch before the storyboard meeting, but should be back soon | 17:14 |
| *** irclogbot_3 has joined #openstack-infra | 17:15 | |
| clarkb | yes I ran the script at https://review.opendev.org/#/c/671102/ which runs mkswap | 17:15 |
| openstackgerrit | Merged opendev/system-config master: Silence InsecureRequestWarning and password warning https://review.opendev.org/671000 | 17:15 |
| *** electrofelix has quit IRC | 17:15 | |
| *** dtantsur is now known as dtantsur|afk | 17:16 | |
| *** ociuhandu has joined #openstack-infra | 17:17 | |
| *** irclogbot_3 has quit IRC | 17:18 | |
| *** irclogbot_1 has joined #openstack-infra | 17:19 | |
| clarkb | http://paste.openstack.org/show/754517/ it seems to hae done what I asked it and manually running mount -a reports no errors and has a 0 exit code so I wonder if I want to do this after the boot process if swapon is required? | 17:19 |
| *** jamesmcarthur has joined #openstack-infra | 17:19 | |
| clarkb | in which case a reboot just to double check it all works may be worthwhile (and we'll reboot during our regular launch node process too) | 17:19 |
| clarkb | I'm going to disable gitea06 in the lb temporarily and reboot | 17:20 |
| *** ociuhandu has quit IRC | 17:22 | |
| fungi | oh, yeah i think you have to swapon if you don't reboot | 17:22 |
| *** irclogbot_1 has quit IRC | 17:22 | |
| fungi | swapon -a will happen at boot and engage all the swap partitions listed in fstab | 17:22 |
| fungi | mount -a does nothing with swap afaik | 17:22 |
| *** irclogbot_0 has joined #openstack-infra | 17:23 | |
| fungi | (because swap partitions aren't "mounted" in the traditional filesystem sense) | 17:23 |
| fungi | anyway, lunch. bbiaw | 17:23 |
| clarkb | oh right | 17:23 |
| clarkb | also reading man pages apparently linux prefers we don't use fallocate craeted files for this | 17:24 |
| clarkb | maybe I should start over and use a dd'd file | 17:24 |
| *** irclogbot_0 has quit IRC | 17:26 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 17:26 |
| *** chandankumar is now known as raukadah | 17:26 | |
| corvus | mordred: i set an autohold on that so we can try to figure out why puppet isn't doing anything | 17:26 |
| *** irclogbot_1 has joined #openstack-infra | 17:27 | |
| *** irclogbot_1 has quit IRC | 17:30 | |
| *** irclogbot_1 has joined #openstack-infra | 17:31 | |
| *** weifan has joined #openstack-infra | 17:32 | |
| *** irclogbot_1 has quit IRC | 17:34 | |
| *** irclogbot_1 has joined #openstack-infra | 17:35 | |
| mordred | cool | 17:35 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use swapfile if no extra device is present https://review.opendev.org/671102 | 17:38 |
| clarkb | infra-root ^ that worked and gitea06 now has swap | 17:38 |
| *** irclogbot_1 has quit IRC | 17:38 | |
| *** irclogbot_3 has joined #openstack-infra | 17:39 | |
| *** goldyfruit has quit IRC | 17:39 | |
| *** goldyfruit has joined #openstack-infra | 17:40 | |
| corvus | clarkb: new project approved | 17:40 |
| openstackgerrit | Merged openstack/pbr master: Add Python 3 Train unit tests https://review.opendev.org/669467 | 17:41 |
| *** irclogbot_3 has quit IRC | 17:42 | |
| *** irclogbot_1 has joined #openstack-infra | 17:43 | |
| AJaeger | team, I now have a "Toggle Extra CI" button at the bottom of https://review.opendev.org/#/q/is:watched+is:open ;( | 17:43 |
| clarkb | AJaeger: what browser? I don't have that in firefox | 17:44 |
| clarkb | AJaeger: maybe try a hard refresh? | 17:45 |
| AJaeger | clarkb: that is after refresh - firefox. But let me try again... | 17:45 |
| AJaeger | clarkb: can you try hard refresh, please? | 17:45 |
| clarkb | fwiw I tried a hard rfresh too in case I had bad caches but doesn't happen | 17:45 |
| AJaeger | mmh, interesting | 17:45 |
| clarkb | let me try in a new browser tab group thing | 17:45 |
| *** irclogbot_1 has quit IRC | 17:46 | |
| clarkb | ya doesn't happen for me there either | 17:46 |
| * AJaeger experiments a bit as well now | 17:46 | |
| *** irclogbot_3 has joined #openstack-infra | 17:47 | |
| Shrews | i have that button too, fwiw | 17:47 |
| Shrews | does nothing when pressed | 17:47 |
| AJaeger | clarkb: maybe I have other repos in my search? It still shows up | 17:47 |
| * AJaeger restarted firefox | 17:48 | |
| clarkb | AJaeger: oh ya it could be related to the results | 17:48 |
| AJaeger | clarkb: try https://review.opendev.org/#/q/topic:retire-api-site | 17:48 |
| clarkb | not there either. I see it flash by on a refresh but it goes away | 17:49 |
| Shrews | oh, i only see it on reviews, not the searches | 17:49 |
| AJaeger | interesting, does not happen in chromium | 17:50 |
| *** irclogbot_3 has quit IRC | 17:50 | |
| AJaeger | I see two flash by - and then one in the end. Might be that hard reloading does not work correctly ;( | 17:50 |
| *** irclogbot_1 has joined #openstack-infra | 17:51 | |
| openstackgerrit | Merged openstack/project-config master: Add microstack project. https://review.opendev.org/668681 | 17:52 |
| *** ociuhandu has joined #openstack-infra | 17:53 | |
| AJaeger | clarkb, Shrews, I removed cache and cookie - now it's normal again. | 17:54 |
| *** irclogbot_1 has quit IRC | 17:54 | |
| *** eernst has joined #openstack-infra | 17:54 | |
| *** irclogbot_1 has joined #openstack-infra | 17:55 | |
| clarkb | I'm guessing you cached the broken state | 17:57 |
| clarkb | and that wasn't doing the right thing on those pages | 17:57 |
| *** irclogbot_1 has quit IRC | 17:58 | |
| AJaeger | yeah ;/ | 17:58 |
| *** irclogbot_2 has joined #openstack-infra | 17:59 | |
| *** ociuhandu has quit IRC | 18:01 | |
| *** ociuhandu_ has joined #openstack-infra | 18:01 | |
| *** irclogbot_2 has quit IRC | 18:02 | |
| *** irclogbot_0 has joined #openstack-infra | 18:03 | |
| *** dims has joined #openstack-infra | 18:05 | |
| *** tdasilva has quit IRC | 18:06 | |
| *** irclogbot_0 has quit IRC | 18:06 | |
| *** irclogbot_3 has joined #openstack-infra | 18:07 | |
| *** rh-jelabarre has quit IRC | 18:10 | |
| *** irclogbot_3 has quit IRC | 18:10 | |
| *** rkukura_ has joined #openstack-infra | 18:10 | |
| *** irclogbot_2 has joined #openstack-infra | 18:11 | |
| *** rkukura has quit IRC | 18:12 | |
| *** rkukura_ is now known as rkukura | 18:12 | |
| *** irclogbot_2 has quit IRC | 18:14 | |
| *** irclogbot_1 has joined #openstack-infra | 18:15 | |
| *** rh-jelabarre has joined #openstack-infra | 18:15 | |
| corvus | PLAY [Create repos on gitea servers] ******************************************* | 18:16 |
| corvus | fingers crossed | 18:17 |
| corvus | https://opendev.org/x/microstack exists | 18:17 |
| corvus | waiting on gerrit now | 18:17 |
| *** irclogbot_1 has quit IRC | 18:18 | |
| *** ociuhandu_ has quit IRC | 18:18 | |
| mordred | corvus: I believe in the puppet | 18:18 |
| *** irclogbot_2 has joined #openstack-infra | 18:19 | |
| AJaeger | ;) | 18:19 |
| corvus | m-p is running | 18:20 |
| AJaeger | config-core, here're two changes for repo retirement if you there's interest: https://review.opendev.org/671268 and https://review.opendev.org/670058 | 18:20 |
| corvus | https://opendev.org/x/microstack is imported | 18:20 |
| AJaeger | sorry, meant https://review.opendev.org/671263 instead of 671268 | 18:20 |
| corvus | clarkb, mordred, AJaeger: i think that means the new project creation stuff is working | 18:20 |
| mordred | corvus: \o/ | 18:21 |
| corvus | we're 2 steps away from having that run as a post-merge job :) | 18:21 |
| * mordred does a little dance | 18:21 | |
| AJaeger | corvus: excellent! | 18:21 |
| corvus | jamespage: https://opendev.org/x/microstack is in place | 18:21 |
| AJaeger | jamespage: we imported also the new branch https://opendev.org/x/microstack/src/branch/configure-hook - did you really want that? | 18:22 |
| *** irclogbot_2 has quit IRC | 18:22 | |
| *** irclogbot_1 has joined #openstack-infra | 18:23 | |
| *** ramishra has quit IRC | 18:25 | |
| *** irclogbot_1 has quit IRC | 18:26 | |
| *** eernst has quit IRC | 18:27 | |
| *** irclogbot_2 has joined #openstack-infra | 18:27 | |
| *** crodriguez has joined #openstack-infra | 18:30 | |
| *** irclogbot_2 has quit IRC | 18:30 | |
| mordred | corvus: the gitea PR was merged upstream | 18:31 |
| *** irclogbot_3 has joined #openstack-infra | 18:31 | |
| mordred | corvus: so I think it would be safe to backport it into the corvus branch we build gitea from - or we can pick it up later | 18:31 |
| *** irclogbot_3 has quit IRC | 18:34 | |
| *** irclogbot_1 has joined #openstack-infra | 18:35 | |
| *** e0ne has joined #openstack-infra | 18:36 | |
| *** irclogbot_1 has quit IRC | 18:38 | |
| *** irclogbot_1 has joined #openstack-infra | 18:39 | |
| fungi | okay, back | 18:41 |
| *** irclogbot_1 has quit IRC | 18:42 | |
| fungi | clarkb: yeah, swapping to sparse files is "not good[tm]" | 18:42 |
| *** irclogbot_1 has joined #openstack-infra | 18:43 | |
| fungi | and while i guess fallocate doesn't create sparse files, it's also problematic since the blocks still need to be initialized on demand | 18:43 |
| fungi | which means a bit of extra overhead on first use | 18:43 |
| *** irclogbot_1 has quit IRC | 18:46 | |
| *** irclogbot_3 has joined #openstack-infra | 18:47 | |
| *** mattw4 has quit IRC | 18:47 | |
| *** irclogbot_3 has quit IRC | 18:50 | |
| *** irclogbot_2 has joined #openstack-infra | 18:51 | |
| *** weifan has quit IRC | 18:52 | |
| *** e0ne has quit IRC | 18:52 | |
| *** irclogbot_2 has quit IRC | 18:54 | |
| *** aaronsheffield has quit IRC | 18:54 | |
| *** irclogbot_1 has joined #openstack-infra | 18:55 | |
| *** ianychoi has quit IRC | 18:56 | |
| *** ianychoi has joined #openstack-infra | 18:56 | |
| *** irclogbot_1 has quit IRC | 18:58 | |
| *** bhavikdbavishi has quit IRC | 18:58 | |
| *** irclogbot_1 has joined #openstack-infra | 18:59 | |
| *** irclogbot_1 has quit IRC | 19:02 | |
| *** irclogbot_2 has joined #openstack-infra | 19:03 | |
| *** ralonsoh has quit IRC | 19:04 | |
| *** irclogbot_2 has quit IRC | 19:06 | |
| *** irclogbot_1 has joined #openstack-infra | 19:07 | |
| mnaser | would it be okay if we temporarily banned irclogbot ? | 19:07 |
| fungi | who/what/where is irclogbot? | 19:09 |
| fungi | oh, i just toggled my weechat smart filters | 19:09 |
| fungi | i guess the join/partspam is what you're referring to? | 19:09 |
| fungi | just this channel, or is it being a problem in other channels you're aware of? | 19:10 |
| *** irclogbot_1 has quit IRC | 19:10 | |
| *** irclogbot_3 has joined #openstack-infra | 19:11 | |
| clarkb | fungi: re swap stuff https://review.opendev.org/#/c/671102/ is ready for review if you want take a look at it. Tested on gitea06 | 19:14 |
| *** irclogbot_3 has quit IRC | 19:14 | |
| *** irclogbot_0 has joined #openstack-infra | 19:15 | |
| *** ociuhandu has joined #openstack-infra | 19:15 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66750&rra_id=all cacti is aware too | 19:16 |
| clarkb | mordred corvus: that is exciting | 19:17 |
| * clarkb got nerd sniped a bit reading up on china visa stuff today | 19:17 | |
| clarkb | fungi: yesterday I had mentioned being willing to help with the wiki git repo puppet weirdness. Anything I can look at there? | 19:17 |
| *** irclogbot_0 has quit IRC | 19:18 | |
| *** igordc has quit IRC | 19:18 | |
| *** irclogbot_2 has joined #openstack-infra | 19:19 | |
| clarkb | fungi: mnaser you may want to reach out to edleafe too? | 19:20 |
| mnaser | fungi, clarkb: yeah it's been taking up a lot of my buffer | 19:20 |
| mnaser | i pinged them in #openstack-dev but no response | 19:20 |
| clarkb | but ya I've completely missed it beacuse of smartfilters | 19:20 |
| *** ociuhandu has quit IRC | 19:21 | |
| *** tdasilva has joined #openstack-infra | 19:21 | |
| *** irclogbot_2 has quit IRC | 19:22 | |
| *** irclogbot_2 has joined #openstack-infra | 19:23 | |
| *** rh-jelabarre has quit IRC | 19:25 | |
| mnaser | oh til about smart filtter | 19:26 |
| mnaser | ill enable it i guess | 19:26 |
| *** irclogbot_2 has quit IRC | 19:26 | |
| mnaser | ah | 19:26 |
| mnaser | much better. | 19:26 |
| *** irclogbot_1 has joined #openstack-infra | 19:27 | |
| fungi | mnaser: that said, you're an irc admin for our channels too. if you deem it necessary to block a user, bookmark https://docs.openstack.org/infra/system-config/irc.html#banning-disruptive-users | 19:28 |
| fungi | it's a one-liner away | 19:29 |
| mnaser | fungi: cool, i wanted to just bring it up in a bit of a broader group before i took any action (as it wasn't that disruptive at least) | 19:29 |
| mnaser | annoying but not disruptive :p) | 19:29 |
| * mnaser remembers when we had to do deal with spam | 19:29 | |
| fungi | mnaser: as for weechat smart filters, yeah, i don't see join/part or nick change messages outside a 1-minute window from that user sending actual content (unless i toggle the filter off with alt+=) | 19:30 |
| *** irclogbot_1 has quit IRC | 19:30 | |
| mnaser | if anything honestly i never check join/part msgs anyways, i type the nick, hit tab, and i don't get autocomplete then they're gone :p | 19:30 |
| fungi | heh, as it should be! | 19:30 |
| *** irclogbot_3 has joined #openstack-infra | 19:31 | |
| *** rh-jelabarre has joined #openstack-infra | 19:31 | |
| *** irclogbot_3 has quit IRC | 19:34 | |
| *** irclogbot_1 has joined #openstack-infra | 19:35 | |
| *** weifan has joined #openstack-infra | 19:35 | |
| *** kopecmartin is now known as kopecmartin|off | 19:36 | |
| *** irclogbot_1 has quit IRC | 19:38 | |
| *** irclogbot_0 has joined #openstack-infra | 19:39 | |
| clarkb | infra-root I'm looking at gitea logs around the time of the most recent OOM on gitea06. There are a lot of POSTs to git-upload-pack urls whcih I think is what happens when you clone/fetch | 19:41 |
| clarkb | it almost looks liek what we do when dib builds images which is go through all the repos in quick succession and update them (and clone as necessary) | 19:42 |
| *** irclogbot_0 has quit IRC | 19:42 | |
| fungi | yeah, i would not be surprised at all if we're the source of our own denial of service attack there | 19:43 |
| *** irclogbot_3 has joined #openstack-infra | 19:43 | |
| fungi | but also entirely possible this is one or more third-party ci systems slamming us at just the wrong time | 19:43 |
| clarkb | I haven't fully tracked it back to dib beacuse the requests all come from the haproxy so I need to trace it back through there | 19:44 |
| *** irclogbot_3 has quit IRC | 19:46 | |
| *** weifan has quit IRC | 19:46 | |
| *** irclogbot_0 has joined #openstack-infra | 19:47 | |
| *** kaiokmo has quit IRC | 19:48 | |
| *** gyee has quit IRC | 19:49 | |
| *** gyee has joined #openstack-infra | 19:50 | |
| *** irclogbot_0 has quit IRC | 19:50 | |
| *** irclogbot_0 has joined #openstack-infra | 19:51 | |
| clarkb | I'm not sure we are logging connections on the new haproxy like we did on the cgit haproxy | 19:52 |
| clarkb | at least they aren't in docker logs | 19:52 |
| clarkb | checkiing journalctl | 19:52 |
| clarkb | corvus: mordred ^ have you interacted with haproxy logs in this setup yet? | 19:53 |
| corvus | i don't see any log config in haproxy.cfg | 19:53 |
| clarkb | log 127.0.0.1 local0 <- is what we've got in there | 19:53 |
| corvus | oh how'd i miss that | 19:53 |
| corvus | then i bet it's configured to log to syslog and i have no idea if it makes it across the container boundary | 19:54 |
| clarkb | ya | 19:54 |
| *** irclogbot_0 has quit IRC | 19:54 | |
| clarkb | let me exec into the container and see if there is stuff within the container | 19:54 |
| clarkb | we may also need to bump up verbosity? | 19:54 |
| corvus | there is no /dev/log in the container | 19:54 |
| *** irclogbot_1 has joined #openstack-infra | 19:55 | |
| clarkb | and no /var/log/syslog | 19:55 |
| clarkb | maybe we tweak it to go to stdout/err so that docker grabs it? | 19:55 |
| corvus | yeah, or we can bind-mount in /var/log/something | 19:56 |
| corvus | and send it to a file (and set up rotation) | 19:56 |
| * clarkb reads up on haproxy configs | 19:56 | |
| clarkb | I like the consistency of using docker log for all of our services (we could consistently have them log to files though) | 19:56 |
| corvus | yeah, i'm still just fuzzy on the rotation aspect of that | 19:57 |
| corvus | (and retention) | 19:57 |
| corvus | i think i would like it if i knew a solid answer about that :) | 19:57 |
| corvus | cause i don't really like manually setting up log rotation rules | 19:58 |
| clarkb | fwiw the mysql image on dockerhub has excellent docs for adminy stuff like this, haproxy does now | 19:58 |
| clarkb | s/now/not/ | 19:58 |
| *** irclogbot_1 has quit IRC | 19:58 | |
| *** irclogbot_3 has joined #openstack-infra | 19:59 | |
| clarkb | https://github.com/dockerfile/haproxy/issues/3 heh and now I see why | 19:59 |
| mordred | corvus: well, once we have k8s in the mix, I'm fairly certain it *does* rotate the docker logs (because I think we wanted ot see something that had been rotated out before) ... | 20:00 |
| mordred | but I'd love to actually understand the whole story | 20:01 |
| clarkb | corvus: one suggestion in that issue is to mount /dev/log into the container | 20:01 |
| corvus | clarkb: that seems like an easy win | 20:01 |
| clarkb | haproxy does not write to stdout/stderr (for performance reasons) so we'd have to do a sidecar for that anyway which takes syslog and writes to stdout | 20:01 |
| clarkb | I think mounting /dev/log is simple and easy and solves the problem | 20:01 |
| clarkb | and that should already be rotated for us? we might need to modify syslog config a bit but ya should be easy | 20:02 |
| *** irclogbot_3 has quit IRC | 20:02 | |
| mordred | although they did indicate that systemd doesn't associate the output with haproxy if we do that - but maybe that's fine | 20:02 |
| mordred | https://github.com/dockerfile/haproxy/issues/3#issuecomment-89163128 | 20:02 |
| corvus | mordred: is puppet perhaps not doing anything because "ubuntu-xenial-rax-iad-0009040637" is not "review01.opendev.org" ? | 20:03 |
| *** irclogbot_2 has joined #openstack-infra | 20:03 | |
| *** weifan has joined #openstack-infra | 20:03 | |
| clarkb | mordred: we aren't using systemd units for this anyway | 20:03 |
| mordred | clarkb: almost certainly | 20:03 |
| mordred | gah | 20:03 |
| mordred | corvus: almost certainly | 20:03 |
| corvus | mordred: neat; i guess we haven't really addressed running anything with a puppet node specifier in this system yet. | 20:04 |
| *** mattw4 has joined #openstack-infra | 20:04 | |
| *** mattw4 has quit IRC | 20:04 | |
| clarkb | looking at bionic's rsyslog config I think if we just bind mount in /dev/log we'll get the logs in /var/log/syslog | 20:04 |
| corvus | (we do run some puppet, but it's all based on operating system facts, not hostnames) | 20:04 |
| clarkb | we can split them out if we want to, but that doesn't seem necessary at this point | 20:04 |
| *** mattw4 has joined #openstack-infra | 20:04 | |
| clarkb | I'll push up a patch to do the bindmount | 20:05 |
| corvus | clarkb: we collect syslog in the job, so it'll be fully tested | 20:05 |
| clarkb | awesome | 20:05 |
| *** irclogbot_2 has quit IRC | 20:06 | |
| corvus | mordred: maybe we can just run "hostname ..." in the setup playbook we run before we run the service playbook | 20:07 |
| *** irclogbot_1 has joined #openstack-infra | 20:07 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Collect haproxy logs via syslog https://review.opendev.org/671357 | 20:07 |
| clarkb | might be as simple as that? but I think that is the next step in debugging the gitea memory issues | 20:07 |
| mnaser | i forgot is the setup haproxy => gitea | 20:07 |
| mnaser | or haproxy => ?? some http server ?? => gitea ? | 20:07 |
| clarkb | mnaser: haproxy -> gitea | 20:07 |
| mnaser | does gitea not parse x-forwarded-for stuff? | 20:08 |
| mnaser | or is haproxy not sending them | 20:08 |
| clarkb | mnaser: I don't think haproxy is sending them beacuse it is only forwarding tls | 20:08 |
| mnaser | oh so haproxy is doing direct tcp | 20:08 |
| mnaser | ok | 20:08 |
| mnaser | fyi | 20:09 |
| mnaser | https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt | 20:09 |
| mnaser | (i only recently learned it existed) | 20:09 |
| *** irclogbot_1 has quit IRC | 20:09 | |
| corvus | neat -- so if we had apache on the gitea hosts, we could do that and get good logging from apache | 20:10 |
| corvus | (we could also implement that in gitea directly) | 20:11 |
| *** bnemec has quit IRC | 20:11 | |
| corvus | https://github.com/pires/go-proxyproto | 20:11 |
| *** irclogbot_2 has joined #openstack-infra | 20:11 | |
| corvus | implementing that directly in gitea sounds like a good idea | 20:12 |
| mordred | yeah | 20:13 |
| mordred | and not too terribly hard | 20:13 |
| clarkb | ya if macaron was logging: GET /foo for addr via addr that would be great | 20:13 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 20:14 |
| *** irclogbot_2 has quit IRC | 20:14 | |
| corvus | mordred: ^ how's that look? | 20:14 |
| corvus | mordred: i ran 'hostname review01.opendev.org' on the held node, and running puppet has progressed to Error: Function lookup() did not find a value for the name 'gerrit_github_token' | 20:15 |
| *** bnemec has joined #openstack-infra | 20:15 | |
| mordred | \o/ | 20:15 |
| *** igordc has joined #openstack-infra | 20:15 | |
| mordred | corvus: wasnt' there a time when we had ansible (or something) set hostname for us? | 20:15 |
| *** beekneemech has joined #openstack-infra | 20:16 | |
| mordred | I feel like we've done that and not done that back and forth more than once now | 20:16 |
| corvus | mordred: i think maybe we only do it in the launch-node script? | 20:16 |
| *** bnemec has quit IRC | 20:16 | |
| *** beekneemech has quit IRC | 20:16 | |
| mordred | yeah, maybe that's it | 20:16 |
| *** bnemec has joined #openstack-infra | 20:16 | |
| mordred | maybe we should run the ansible that launch-node runs as part of the run-base | 20:17 |
| corvus | but yes, i'm pretty sure we have gone through all the possible permutations of whether, where, and how to set hostnames. at least once. | 20:17 |
| corvus | mordred: maybe? but i think i only want to touch hostnames in base if something's broken :) | 20:17 |
| mordred | yeah. I don't think we should update base | 20:17 |
| *** beekneemech has joined #openstack-infra | 20:18 | |
| corvus | mordred, clarkb: do we have a way to do fake puppet hiera stuff in the ansible-run jobs? | 20:19 |
| clarkb | corvus: I think it currently relies on the defaults we have in the hiera lookups | 20:20 |
| clarkb | well that and the public hiera | 20:20 |
| corvus | clarkb: yeah, that's the conclusion i'm coming to as well; so nothing for private hiera; either defaults or public hiera | 20:20 |
| *** goldyfruit has quit IRC | 20:20 | |
| clarkb | we should be able to use fake host vars / group vars | 20:20 |
| clarkb | since all of that should be handled by the ansible puppet module | 20:21 |
| corvus | so my next question is: should we invent that, or should we draw the line here and say let's just omit gerrit from this job until it's ported over to ansible? | 20:21 |
| corvus | clarkb: oh, okay, it may not be that hard then? | 20:21 |
| mordred | corvus: I'm leaning towards "let's just omit gerrit until it's ported over to ansible" | 20:21 |
| *** xek has quit IRC | 20:21 | |
| mordred | oh. nevermind | 20:21 |
| clarkb | corvus: ya I don't think it would be too hard since we already have the fake host/group vars stuff in place | 20:21 |
| mordred | don't listen to me, listen to clarkb | 20:21 |
| clarkb | and the ansible-puppet role should just automatically use that? | 20:21 |
| corvus | i listen to both :) | 20:22 |
| clarkb | this is the upside to doing hiera as ansible vars | 20:22 |
| mordred | clarkb: that's an excellent point | 20:22 |
| mordred | ++ | 20:22 |
| corvus | okay, how about i give that a try, and if it's significantly more complicated than that, we drop it | 20:22 |
| corvus | (if it's easy enough, then, honestly, it's not wasted work -- it'll be valuable when we port it to ansible; it'll even act as a regression test) | 20:23 |
| *** ociuhandu has joined #openstack-infra | 20:24 | |
| clarkb | ++ | 20:24 |
| mordred | ++ | 20:24 |
| corvus | (ansible group name: gerrit; puppet group name: review) | 20:28 |
| corvus | cool, based on my manual tests on the held node, this should work | 20:28 |
| corvus | is openstackwatch still running? | 20:32 |
| fungi | clarkb: so reading the haproxy dockerfile issue, haproxy 1.9 can log to normal docker logging facilities but if we turn on request logging with that we may end up with performance issues when writes to stdout overrun the buffer and block? | 20:33 |
| fungi | (it was a long read) | 20:33 |
| clarkb | corvus: I awnt to say no? | 20:33 |
| *** ociuhandu has quit IRC | 20:33 | |
| clarkb | fungi: yes | 20:33 |
| fungi | corvus: i believe openstackwatch can go away. i don't even remember the swift url for the rss feed | 20:35 |
| fungi | but pretty sure it's dead | 20:35 |
| *** gfidente is now known as gfidente|afk | 20:36 | |
| *** e0ne has joined #openstack-infra | 20:38 | |
| corvus | okay, we're not going to get the gerrit puppet to do an initial bootstrap without a mysql server, and that's currently trove, right? | 20:39 |
| clarkb | ok I need to make a few more haproxy config changes to make this syslogging work ps in a sec | 20:39 |
| mordred | corvus: yes, that's right. | 20:40 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Collect haproxy logs via syslog https://review.opendev.org/671357 | 20:41 |
| corvus | so our choices there are: make a playbook to set up a mysql server (on a new host or in a container) and substitute its connection info into the hostvars for this job, then throw that away when we upgrade gerrit. or, just shelve this until we upgrade gerrit to one that doesn't need a db. | 20:43 |
| clarkb | aiui there is still a db with gerrit even in latest version? its just used for far less? | 20:44 |
| corvus | i'm happy to do either. i don't think it'd be too much work for option 1; otoh, the gerrit probably doesn't need much testing until option 2 happens.... | 20:44 |
| clarkb | but ya maybe that is indication that we've gotten far enough down the road of testing that for now | 20:44 |
| *** irclogbot_1 has joined #openstack-infra | 20:44 | |
| corvus | somewhere there was an etherpad where someone was working on what we actually need to do to upgrade | 20:44 |
| clarkb | we could h2 it maybe? I bet our puppet doesn't do that though | 20:44 |
| donnyd | So my timeouts are down to just a couple jobs now... for some reason this one just won't run on FN | 20:44 |
| donnyd | message:"RUN END RESULT_TIMED_OUT [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]" | 20:44 |
| clarkb | monty was looking at how to upgrade it using docker images | 20:44 |
| clarkb | donnyd: the tripleo jobs may be right up against the timeout limit already so any variance tips it over? mwhahaha and EmilienM and zbr may know best | 20:45 |
| corvus | someone that wasn't us was looking into it | 20:45 |
| clarkb | corvus: ah | 20:45 |
| donnyd | thanks clarkb | 20:45 |
| mwhahaha | which job/patch? | 20:45 |
| donnyd | Seems like pretty much all the 3O jobs | 20:46 |
| donnyd | http://logstash.openstack.org/#/dashboard/file/logstash.json | 20:46 |
| clarkb | donnyd: have a link to the log file? | 20:46 |
| *** irclogbot_1 has quit IRC | 20:46 | |
| donnyd | http://logs.openstack.org/21/670821/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/d6c6dac/job-output.txt | 20:46 |
| donnyd | http://logs.openstack.org/13/562213/24/check/tripleo-ci-centos-7-containers-multinode/016bb4e/job-output.txt | 20:46 |
| clarkb | fungi: nova lets you attach a cdrom/dvd iso? | 20:47 |
| donnyd | http://logs.openstack.org/97/631497/23/check/tripleo-ci-centos-7-containers-multinode-pike/725bbd5/job-output.txt | 20:47 |
| donnyd | clarkb: thats what a config drive is | 20:47 |
| *** pcaruana has quit IRC | 20:48 | |
| clarkb | donnyd: config drive can also use fat32 disk. But in this case fungi was saying people mount an installer iso and then take a snapshot to make images | 20:48 |
| clarkb | and I didn't think nova let you do that | 20:48 |
| * mwhahaha is unsure the context of this conversation | 20:48 | |
| donnyd | sure can | 20:48 |
| corvus | https://etherpad.openstack.org/p/gerrit-upgrade | 20:48 |
| clarkb | mwhahaha: sorry there are two conversations, the one I pinged you about was jobs timing out on the new fortnebula cloud | 20:48 |
| donnyd | you can also attach a cinder volume, and then install to that and turn it back around and move the volume to glance | 20:48 |
| *** bnemec has quit IRC | 20:48 | |
| clarkb | mwhahaha: donnyd was wanting to look into why that might be happening but they are tripleo jobs and figured tripleo might do a better job of debugging that | 20:49 |
| mwhahaha | clarkb: if those are the logs, it looks like it ran long | 20:49 |
| donnyd | well if you snapshot the installer, you are just getting a snapshot of the ISO | 20:49 |
| mwhahaha | they look to be timing out in the tempest run which means the overall process took a while. might be transit/host io. not exactly clear | 20:50 |
| donnyd | mwhahaha: I am trying to figure out why only the tripleo jobs are the ones not making it before the timer hits | 20:50 |
| corvus | clarkb, mordred: "Before upgrade to 3.0, need to fully migrate off ReviewDb" from https://etherpad.openstack.org/p/gerrit-upgrade -- that seems to say that by 3.0 we have changes, users, and groups in notedb. is something left? web sessions? | 20:50 |
| clarkb | corvus: the not reviewdb second db /me looks up its name | 20:50 |
| donnyd | It previously was I/O, but they are all on dedicated NVME drives now and still not making the cut... So I am not sure if its cycles that its bound by or not | 20:51 |
| clarkb | corvus: accountPatchReviewDb | 20:51 |
| *** jamesmcarthur has quit IRC | 20:51 | |
| *** bnemec has joined #openstack-infra | 20:51 | |
| corvus | clarkb: ugh. | 20:51 |
| mordred | yeah - but I think that's small enough that they say you can just use the builtin db thing | 20:52 |
| corvus | https://gerrit-documentation.storage.googleapis.com/Documentation/3.0.1/config-gerrit.html#accountPatchReviewDb | 20:52 |
| corvus | looks like that's to store whether a user has reviewed files in patches; it could be a lot of small updates. do you think h2 will scale? | 20:53 |
| donnyd | mwhahaha: Also I am not sure its an ipv6 thing, because FN is ipv6 only for inbound | 20:53 |
| mordred | absolutely no idea | 20:53 |
| clarkb | corvus: no we use mysql | 20:53 |
| clarkb | because ya h2 was reported to not scale at all | 20:53 |
| clarkb | at least on the current gerrit | 20:53 |
| mwhahaha | donnyd: so we have ara for our execution in the logs dir, migth help with durations: http://logs.openstack.org/97/631497/23/check/tripleo-ci-centos-7-containers-multinode-pike/725bbd5/logs/ara_oooq/ara-report/ | 20:54 |
| corvus | okay, so gerrit 3.0+ needs a mysql for that | 20:54 |
| *** beekneemech has quit IRC | 20:54 | |
| corvus | clarkb, mordred: updated etherpad with that note | 20:54 |
| mordred | I *know* mysql can handle those. I wish there was a way to just disable that feature - it's mostly neat, but I'm not sure it's worth the trouble of running a db for. oh well. | 20:55 |
| mwhahaha | donnyd: container fetching, are you mirroring? | 20:55 |
| *** tjgresha_nope has quit IRC | 20:55 | |
| donnyd | there is a local mirror | 20:56 |
| mwhahaha | how is it performing | 20:56 |
| *** tjgresha has joined #openstack-infra | 20:56 | |
| donnyd | same as any other provider | 20:56 |
| mwhahaha | so for comparison, http://logs.openstack.org/10/501810/21/check/tripleo-ci-centos-7-containers-multinode-pike/45bb0dc/logs/ara_oooq/ara-report/ | 20:56 |
| mwhahaha | preping containers took 28 & 12 mins on the timeout | 20:56 |
| mwhahaha | vs 19&7 | 20:56 |
| mwhahaha | overall everything ran slower | 20:56 |
| mwhahaha | overcloud deploy 1hr vs 52 mins | 20:57 |
| mwhahaha | undercluod install 35 mins vs 28 | 20:57 |
| donnyd | is there a way to see in the job where its pulling containers from? | 20:57 |
| mwhahaha | i think we might be graphing this stuff somewhere, would need to poke weshay | 20:57 |
| corvus | clarkb, mordred: i'll start on some ansible to run a mariadb in a container so we can add it to that job | 20:58 |
| mwhahaha | donnyd: we pull from docker.io | 20:58 |
| weshay | ya.. docker.io | 20:58 |
| fungi | clarkb: you have to attach a volume, i think, and then reboot? i've seen folks discuss doing it, i've never tried it myself | 20:58 |
| weshay | looking for perf info? | 20:58 |
| mwhahaha | weshay: yea new cloud, trying to see perf numbers | 20:58 |
| clarkb | weshay: mwhahaha are you going through our docker hub proxy in the region or direct to docker? | 20:58 |
| weshay | k.. sec | 20:59 |
| donnyd | fungi: You have to have a volume attached, install to it, and then move the image from cinder to glance | 20:59 |
| weshay | not sure if we have exactly what ur looking for | 20:59 |
| mwhahaha | clarkb: should be your docker hub proxy | 20:59 |
| corvus | "our" :) | 20:59 |
| fungi | donnyd: ahh, yep, so other way around. installer uploaded to glance and tell nova to boot from that. got it | 20:59 |
| weshay | mwhahaha http://dashboard-ci.tripleo.org/d/FEdraO0ik/jobs-exploration?orgId=1&fullscreen&panelId=2 | 20:59 |
| donnyd | yea, you upload the iso to glance, create instance with volume, iso install to volume, upload volume to glance | 21:00 |
| donnyd | boom, iso installs on openstack | 21:00 |
| weshay | the far right one is vexhost 3rd party.. can be ignored | 21:00 |
| mwhahaha | http://logs.openstack.org/97/631497/23/check/tripleo-ci-centos-7-containers-multinode-pike/725bbd5/logs/undercloud/etc/docker/daemon.json.txt.gz | 21:01 |
| mwhahaha | we're using http://mirror.regionone.fortnebula.opendev.org:8082/ | 21:01 |
| clarkb | ok that rules out one thing it could've been | 21:01 |
| fungi | a.k.a. "it's not afs" | 21:01 |
| mwhahaha | since everything is slower, i'd say transit | 21:02 |
| mwhahaha | either between vms or external | 21:02 |
| mwhahaha | because udnercloud/overcloud are on different nodes but the overcloud deploy containers from undercloud | 21:02 |
| mwhahaha | also could still beio | 21:02 |
| clarkb | that seemed to be quick during my testing thoguh and its 40Gbe networking iirc | 21:02 |
| clarkb | but maybe we aren't getting the expected divided throughput on the VMs? | 21:02 |
| *** _erlon_ has quit IRC | 21:02 | |
| clarkb | VMs should be gigabit I would expect | 21:02 |
| fungi | this is proxying dockerhub so going over ipv4 i guess? | 21:03 |
| clarkb | fungi: ya its test VM -> ipv6 -> mirror -> ipv4 docker hub | 21:03 |
| fungi | got it | 21:03 |
| mwhahaha | we do have atop results if you want to look at that for the run | 21:04 |
| mwhahaha | http://logs.openstack.org/97/631497/23/check/tripleo-ci-centos-7-containers-multinode-pike/725bbd5/logs/undercloud/var/log/ | 21:04 |
| fungi | anyway, between provider and dockerhub over open internet it's ipv4, so can't blame flaky global v6 routing | 21:04 |
| mwhahaha | or dstat, http://logs.openstack.org/97/631497/23/check/tripleo-ci-centos-7-containers-multinode-pike/725bbd5/logs/undercloud/var/log/extra/dstat.html.gz | 21:04 |
| donnyd | the build times on the dashboard don't look much different from rax | 21:04 |
| donnyd | Its likely my cpus are much older than the other providers | 21:05 |
| *** mattw4 has quit IRC | 21:05 | |
| * fungi wouldn't put money on that | 21:05 | |
| fungi | sucker bet | 21:05 |
| fungi | commercial cloud providers can sometimes be very... "cost-conscious" about their hardware upgrade schedules | 21:06 |
| fungi | i used to work for one | 21:06 |
| mordred | me too | 21:06 |
| mwhahaha | looks to be more disk | 21:07 |
| fungi | mordred: you worked for more than one | 21:07 |
| *** mattw4 has joined #openstack-infra | 21:07 | |
| mwhahaha | compare timed out http://logs.openstack.org/97/631497/23/check/tripleo-ci-centos-7-containers-multinode-pike/725bbd5/logs/undercloud/var/log/extra/dstat.html.gz | 21:07 |
| mwhahaha | tp success http://logs.openstack.org/10/501810/21/check/tripleo-ci-centos-7-containers-multinode-pike/45bb0dc/logs/undercloud/var/log/extra/dstat.html.gz | 21:07 |
| mwhahaha | timedout has less dsk/total and io/total | 21:07 |
| donnyd | well I have them all on NVME only storage, not sure how much faster I can make them | 21:07 |
| fungi | rub them with cheetah blood? | 21:08 |
| clarkb | perhaps there is something with kvm/qemu slowing stuff down? though iirc you said the cache settings were already set to the dangermode | 21:08 |
| donnyd | clarkb: I removed those settings when I moved over to iscsi | 21:09 |
| donnyd | but there is one hypervisor still left on nfs till more gear gets here tomorrow and I can install over the weekend | 21:10 |
| mordred | fungi: either 2 or 3, depending on how you feel about definitions | 21:10 |
| donnyd | maybe that job unluckily gets scheduled on that node | 21:10 |
| fungi | yeah, depending on definitions of "cloud" and "provider" (and more importantly "work") | 21:10 |
| clarkb | donnyd: we should be able to check that, one sec | 21:11 |
| clarkb | I'll get the hostid | 21:11 |
| clarkb | donnyd: 2c7e4af000b57bf5d4efb85fb84fac135554686c15c3b4944518712c for http://logs.openstack.org/13/562213/24/check/tripleo-ci-centos-7-containers-multinode/016bb4e/zuul-info/inventory.yaml which was one of the jobs you listed | 21:13 |
| donnyd | Ok, i will run down that hostid | 21:13 |
| mordred | fungi: yup. all three of those are definitely salient | 21:14 |
| donnyd | well that job def ran on the host with nvme storage | 21:15 |
| *** e0ne has quit IRC | 21:15 | |
| clarkb | donnyd: is the mirror on nvme? | 21:16 |
| donnyd | not yet, this weekend it will be | 21:16 |
| clarkb | could be slowness reading out of the mirror cache in that case | 21:16 |
| donnyd | yea, that is a possibility | 21:17 |
| clarkb | infra-root http://logs.openstack.org/57/671357/2/check/system-config-run-gitea/0085f2e/gitea-lb01.opendev.org/syslog.txt.gz#_Jul_17_21_07_44 it works | 21:18 |
| clarkb | can we get reviews on https://review.opendev.org/#/c/671357/ and then we'll need to restart haproxy once that goes in (maybe? this modifies the docker compose file so maybe it will be automatic) | 21:18 |
| clarkb | thats the next step in debugging the OOMing I think | 21:19 |
| mordred | clarkb: wfm | 21:19 |
| donnyd | ara is pretty slick, are all jobs reporting to it or just 3o | 21:21 |
| mwhahaha | all have it | 21:21 |
| mwhahaha | we just have another run | 21:21 |
| mwhahaha | because our tooling kicks off a different ansible so we capture the tripleo-quickstart stuff sperately | 21:21 |
| *** irclogbot_1 has joined #openstack-infra | 21:23 | |
| fungi | donnyd: it's an optional role but i think we include it for most jobs these days | 21:23 |
| donnyd | clarkb: this weekend FN will be out of service (just hypervisors), should i just scale down the quota for the weekend while its in maintenance. The API's should stay up | 21:24 |
| clarkb | donnyd: yup that should work fine | 21:24 |
| donnyd | Weekend load is pretty low anyways | 21:24 |
| clarkb | set quota to 0, wait for existing jobs to finish, do your work and turn it back up again | 21:25 |
| donnyd | at least last weekend was | 21:25 |
| clarkb | ya weekends are slow | 21:25 |
| *** rosmaita has quit IRC | 21:25 | |
| fungi | donnyd: looks like it's in our base job playbook these days, so basically everything should include an ara report now: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L5 | 21:26 |
| *** rosmaita has joined #openstack-infra | 21:27 | |
| *** joeguo has quit IRC | 21:30 | |
| *** joeguo has joined #openstack-infra | 21:30 | |
| *** tdasilva_ has joined #openstack-infra | 21:31 | |
| *** tdasilva has quit IRC | 21:34 | |
| *** ociuhandu has joined #openstack-infra | 21:35 | |
| *** whoami-rajat has quit IRC | 21:36 | |
| *** betherly has joined #openstack-infra | 21:36 | |
| *** betherly has quit IRC | 21:41 | |
| *** ociuhandu has quit IRC | 21:45 | |
| *** sgw has quit IRC | 21:46 | |
| *** panda has quit IRC | 21:46 | |
| *** ijw has quit IRC | 21:48 | |
| *** panda has joined #openstack-infra | 21:48 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 21:50 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 21:51 |
| openstackgerrit | Merged opendev/system-config master: Collect haproxy logs via syslog https://review.opendev.org/671357 | 21:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Run dstat and generate graphs in unit tests https://review.opendev.org/610100 | 21:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Run dstat and generate graphs in unit tests https://review.opendev.org/610100 | 21:56 |
| *** ociuhandu has joined #openstack-infra | 22:00 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add dstat graph roles https://review.opendev.org/670402 | 22:01 |
| *** ociuhandu has quit IRC | 22:06 | |
| *** dchen has joined #openstack-infra | 22:12 | |
| *** betherly has joined #openstack-infra | 22:14 | |
| clarkb | I think docker compose did just restart haproxy | 22:14 |
| clarkb | and we have request info in syslog now | 22:15 |
| clarkb | I guess wewait for OOMs now | 22:15 |
| *** betherly has quit IRC | 22:18 | |
| fungi | and the confused user reports about git fetch calls with connections reset | 22:19 |
| *** ekultails has quit IRC | 22:20 | |
| *** slaweq has quit IRC | 22:23 | |
| *** pkopec has quit IRC | 22:26 | |
| clarkb | zxiiro: what was the documentation linter yall are using? it is pluggable iirc and has an optional spell checker? | 22:30 |
| zxiiro | clarkb: we've been using this https://github.com/btford/write-good but it's tied to npm. I've since discovered an alternative https://github.com/amperser/proselint/ but I haven't had time to investigate it. | 22:32 |
| zxiiro | proselint works with pre-commit too | 22:32 |
| clarkb | thanks | 22:34 |
| clarkb | "Avoid words banned by the NFL" | 22:37 |
| clarkb | I'm guessing hte list is a good list, but I find it funny that the NFL influences that checker :) | 22:38 |
| *** logan- has quit IRC | 22:38 | |
| zxiiro | Hah, I guess NFL's influence is far reaching :) | 22:38 |
| *** logan- has joined #openstack-infra | 22:39 | |
| *** tesseract has quit IRC | 22:40 | |
| *** eernst_ has joined #openstack-infra | 22:40 | |
| clarkb | the list apparently applies to what you can put on the back of a jersey (because you can buy custom official jerseys" | 22:42 |
| *** eernst_ has quit IRC | 22:43 | |
| *** eernst has joined #openstack-infra | 22:43 | |
| *** eernst_ has joined #openstack-infra | 22:43 | |
| *** eernst has quit IRC | 22:43 | |
| *** eernst_ has quit IRC | 22:47 | |
| *** eernst has joined #openstack-infra | 22:51 | |
| openstackgerrit | Merged zuul/zuul master: Correct git-review install in CentOS 7 quick-start https://review.opendev.org/670736 | 22:51 |
| fungi | probably more inclusive than the list of words banned by the nhl | 22:54 |
| fungi | which is what i usually go by | 22:54 |
| fungi | sometime i want to make a "simplified english" variant which includes only words from the tournament scrabble dictionary | 22:55 |
| * fungi speaks only the queen's scrabbleish | 22:55 | |
| *** eernst has quit IRC | 22:56 | |
| *** eernst has joined #openstack-infra | 22:56 | |
| corvus | i think the opposite would be more fun | 22:56 |
| fungi | only words not included in the scrabble dictionary? ...maybe so | 22:56 |
| fungi | i still look for lots opportunities to use "qi" in a sentence | 22:57 |
| corvus | well, this is weird: http://logs.openstack.org/62/671162/7/check/system-config-run-git/6dc111f/ | 22:57 |
| clarkb | corvus: now zuul knows its "abc"s | 22:58 |
| corvus | clarkb: yes, though omitting the letters "a", "b", and "c". :) | 22:59 |
| corvus | that field is read as an as_list... so that shouldn't have happened | 22:59 |
| corvus | nope missed one | 23:00 |
| *** eernst has quit IRC | 23:00 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Parse nodeset.group.nodes as a list https://review.opendev.org/671372 | 23:02 |
| corvus | that's the prob | 23:02 |
| *** ociuhandu has joined #openstack-infra | 23:02 | |
| corvus | in other news, apparently the 'hostname' ansible module doesn't work on xenial? | 23:03 |
| corvus | http://logs.openstack.org/62/671162/7/check/system-config-run-base/28ee111/ara-report/result/62c2c765-71b2-4c8e-a19b-4311a916be20/ | 23:03 |
| corvus | so i guess i'll just change that to "command: hostname" | 23:03 |
| corvus | (you had one job, hostname module) | 23:03 |
| clarkb | failed to create bus connection? | 23:03 |
| clarkb | is it trying to set the hostname over dbus? | 23:03 |
| *** ricolin_ has joined #openstack-infra | 23:03 | |
| corvus | i assume so | 23:04 |
| corvus | i mean, how else would you run a command on a unix system? | 23:04 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 23:04 |
| clarkb | tumbleweed has apparently rebuilt all the things with link time optimizations | 23:06 |
| *** ricolin has quit IRC | 23:06 | |
| clarkb | so I have to isntall thousands of packages but I save 115 MB and things are supposed to go faster | 23:06 |
| clarkb | hopefully nothing breaks | 23:06 |
| *** ociuhandu has quit IRC | 23:06 | |
| corvus | clarkb: neat! let me know how it goes (if you can) | 23:06 |
| *** rcernin has joined #openstack-infra | 23:11 | |
| *** tkajinam has joined #openstack-infra | 23:15 | |
| *** rcernin has quit IRC | 23:16 | |
| *** rcernin has joined #openstack-infra | 23:17 | |
| *** jtomasek has quit IRC | 23:17 | |
| *** jtomasek has joined #openstack-infra | 23:19 | |
| *** betherly has joined #openstack-infra | 23:21 | |
| *** diablo_rojo has joined #openstack-infra | 23:21 | |
| *** jamesmcarthur has joined #openstack-infra | 23:23 | |
| *** tdasilva_ has quit IRC | 23:25 | |
| *** tdasilva_ has joined #openstack-infra | 23:25 | |
| *** betherly has quit IRC | 23:26 | |
| *** hwoarang has quit IRC | 23:37 | |
| *** hwoarang has joined #openstack-infra | 23:38 | |
| fungi | if we don't hear from you for a while, that's an answer of a sort too | 23:52 |
| *** diablo_rojo has quit IRC | 23:54 | |
| clarkb | heh I actually stepped away | 23:55 |
| *** goldyfruit has joined #openstack-infra | 23:56 | |
| fungi | ;) | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
