| mordred | fungi: precise? | 00:01 |
|---|---|---|
| fungi | mordred: yes | 00:01 |
| fungi | needs upgrading, but waiting until after the summit | 00:01 |
| mordred | oy | 00:01 |
| *** ddieterly has joined #openstack-infra | 00:01 | |
| *** salv-orl_ has joined #openstack-infra | 00:02 | |
| *** eil397 has quit IRC | 00:02 | |
| fungi | i noticed the "no hosts matched" play output earlier too. is that... expected? | 00:02 |
| mordred | noe | 00:03 |
| *** dims_ has joined #openstack-infra | 00:03 | |
| pabelanger | I noticed that too | 00:04 |
| *** dims has quit IRC | 00:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Centos7 wheel cache jobs https://review.openstack.org/290272 | 00:04 |
| *** mylu has quit IRC | 00:05 | |
| *** salv-orlando has quit IRC | 00:05 | |
| openstackgerrit | Merged openstack-infra/system-config: Add note about running keystone on 443 https://review.openstack.org/201667 | 00:05 |
| *** Sukhdev has joined #openstack-infra | 00:06 | |
| *** salv-orl_ has quit IRC | 00:07 | |
| *** heyongli has quit IRC | 00:07 | |
| *** heyongli has joined #openstack-infra | 00:08 | |
| *** baoli has quit IRC | 00:09 | |
| mordred | fungi: ok. so there is a regression in os-client-config | 00:10 |
| *** baoli has joined #openstack-infra | 00:10 | |
| *** netsin has joined #openstack-infra | 00:11 | |
| fungi | nice! | 00:11 |
| mordred | no - actually, not quite | 00:12 |
| mordred | there is a thing it's supposed to do that it's not doing completely | 00:12 |
| mordred | I'll have to think about it slighlty more | 00:12 |
| mordred | workaround comign | 00:12 |
| openstackgerrit | Merged openstack-infra/system-config: Give 0775 to launch-node.py https://review.openstack.org/295959 | 00:14 |
| *** amrith is now known as _amrith_ | 00:15 | |
| *** daemontool has quit IRC | 00:15 | |
| *** rguillebert has quit IRC | 00:17 | |
| *** heyongli has quit IRC | 00:17 | |
| pabelanger | clarkb: manually triggered job now | 00:18 |
| *** heyongli has joined #openstack-infra | 00:18 | |
| clarkb | pabelanger: ok, I +2'd the centos7 mariadb and python34-devel change | 00:18 |
| clarkb | we can worry about fedora if/when we build wheels for it | 00:18 |
| pabelanger | clarkb: great | 00:18 |
| *** achanda has quit IRC | 00:18 | |
| ianw | hmm, looks promising | 00:18 |
| *** Qiming has quit IRC | 00:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Don't pass OS_CLOUD and OS_REGION_NAME to expand-groups https://review.openstack.org/296127 | 00:19 |
| mordred | fungi: ^^ | 00:19 |
| *** mylu has joined #openstack-infra | 00:20 | |
| *** achanda has joined #openstack-infra | 00:21 | |
| fungi | mordred: needs a rebase | 00:21 |
| fungi | probably merge-conflicted with 295959 | 00:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Don't pass OS_CLOUD and OS_REGION_NAME to expand-groups https://review.openstack.org/296127 | 00:22 |
| fungi | the file permissions change | 00:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Don't pass OS_CLOUD and OS_REGION_NAME to expand-groups https://review.openstack.org/296127 | 00:23 |
| *** mylu has quit IRC | 00:25 | |
| *** _amrith_ is now known as amrith | 00:25 | |
| mordred | fungi: ^^ that's rebased and has the change I made in the local dir | 00:25 |
| fungi | saw in screen | 00:25 |
| *** netsin has quit IRC | 00:26 | |
| fungi | that's looking a little more complete than before | 00:26 |
| *** netsin has joined #openstack-infra | 00:26 | |
| *** SumitNaiksatam has quit IRC | 00:26 | |
| *** heyongli has quit IRC | 00:28 | |
| *** sridhar_ram has joined #openstack-infra | 00:28 | |
| *** heyongli has joined #openstack-infra | 00:28 | |
| *** sshnaidm has quit IRC | 00:30 | |
| *** jiggalator has joined #openstack-infra | 00:31 | |
| *** baoli has quit IRC | 00:32 | |
| *** Swami has quit IRC | 00:32 | |
| *** tphummel has quit IRC | 00:33 | |
| *** mriedem has joined #openstack-infra | 00:34 | |
| *** thorst has joined #openstack-infra | 00:34 | |
| *** angdraug has quit IRC | 00:35 | |
| *** thorst_ has joined #openstack-infra | 00:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Don't pass OS_CLOUD and OS_REGION_NAME to expand-groups https://review.openstack.org/296127 | 00:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Also protected against OS env vars in expand-groups https://review.openstack.org/296128 | 00:36 |
| mordred | fungi: k. that should be everything ^^ | 00:36 |
| mordred | fungi: there is a chicken and egg thing here ... which is that if a host we're launching needs group variables to be copied, it will not get them | 00:38 |
| *** jiggalator has quit IRC | 00:38 | |
| mordred | fungi: since it will not have group information when we run ansible because we haven't run expand-groups.sh yet | 00:38 |
| *** heyongli has quit IRC | 00:38 | |
| mordred | however, we don't wnat to do that yet, because the server might be broken and we don't want OTHER ansible runs to pick it up until we're done | 00:38 |
| *** heyongli has joined #openstack-infra | 00:38 | |
| *** tphummel has joined #openstack-infra | 00:39 | |
| *** thorst has quit IRC | 00:40 | |
| *** achanda has quit IRC | 00:40 | |
| fungi | mordred: so how much of a shadow copy of the system is going to be required to be able to use the same tooling to launch new servers while preventing the existing tooling from trying to step on the new servers mid-deployment? | 00:40 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add release notes job for Kolla repo https://review.openstack.org/296129 | 00:40 |
| *** piet has joined #openstack-infra | 00:41 | |
| *** krotscheck has joined #openstack-infra | 00:42 | |
| *** sputnik13 has quit IRC | 00:43 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add verbose to wheel build https://review.openstack.org/296130 | 00:45 |
| ianw | pabelanger: ^ should show compilation logs, which should give us a clue what headers, etc are missing | 00:46 |
| *** amitgandhinz has joined #openstack-infra | 00:46 | |
| mordred | fungi: wel - it gets easier as we collapse things into the same system, actually | 00:46 |
| mordred | fungi: part of the current problem is that we're still spread across multiple things becuase we're moving stepwise into the future | 00:46 |
| *** erlon has quit IRC | 00:47 | |
| mordred | fungi: but I'm going to afk for the night - so I'll write up something about thoughts later | 00:47 |
| fungi | mordred: anyway, this _particular_ server doesn't require any group variables i guess, so it it safe to proceed once we have those in place? will just cherry-picking them into a local clone of system-config be sufficient for now? | 00:47 |
| pabelanger | ianw: oh, neat. I assume you tested it local? | 00:47 |
| pabelanger | or should I try it on centos7 node | 00:47 |
| *** heyongli has quit IRC | 00:48 | |
| *** heyongli has joined #openstack-infra | 00:48 | |
| *** pvaneck has quit IRC | 00:49 | |
| pabelanger | ianw: Ya, tested local. works here | 00:50 |
| ianw | pabelanger: yep, just in a virtualenv | 00:50 |
| pabelanger | clarkb: fungi: mind helping with 296130 ? | 00:50 |
| *** sdake has quit IRC | 00:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:50 | |
| *** sdake has joined #openstack-infra | 00:51 | |
| clarkb | sure, you cna also check ~jenkins/.pip/somethingorother for the logs | 00:51 |
| *** tonytan4_ has quit IRC | 00:52 | |
| pabelanger | clarkb: currently empty on the slave | 00:52 |
| clarkb | huh | 00:53 |
| ianw | pabelanger: maybe it goes in the wheelhouse? | 00:53 |
| ianw | yeah, nah | 00:53 |
| pabelanger | ya, looking | 00:54 |
| ianw | pabelanger: yeah, as i suspected, failed writing to afs | 00:54 |
| pabelanger | ya, see that now | 00:54 |
| pabelanger | let me see what is the issue | 00:54 |
| ianw | k5start didn't seem to fail... | 00:55 |
| pabelanger | maybe we need to create /afs/.openstack.org/mirror/wheel/centos-7-x86_64/ first? | 00:56 |
| pabelanger | cause I don't see it | 00:56 |
| *** okay22many has joined #openstack-infra | 00:56 | |
| ianw | pabelanger: can you "k5start -t -f /etc/wheel.keytab service/wheel -- timeout -k 2m 30m touch /afs/.openstack.org/mirror/foo" ? | 00:57 |
| clarkb | oh if you don't already have a volume yet I think you need that | 00:57 |
| clarkb | not sure if we are sharing a wheel volume for all distros? | 00:57 |
| pabelanger | Right, I think we need to check the ACL on it | 00:57 |
| pabelanger | but not sure how to do that | 00:57 |
| *** heyongli has quit IRC | 00:58 | |
| *** heyongli has joined #openstack-infra | 00:59 | |
| pabelanger | touch: cannot touch ‘/afs/.openstack.org/mirror/foo’: Permission denied | 00:59 |
| pabelanger | so we don't have permission to write to mirror ATM | 00:59 |
| pabelanger | clarkb: do you know how to fix that? | 00:59 |
| *** amitgandhinz has quit IRC | 01:00 | |
| clarkb | pabelanger: I think you likely have to create a wheel/centos-7-x86_64 dir (possibly its own volume if ubuntu has its own volume) | 01:00 |
| clarkb | pabelanger: there are afs docs published on how to get an admin account and all that | 01:00 |
| pabelanger | Right, reading that now | 01:00 |
| openstackgerrit | Merged openstack-infra/project-config: Add verbose to wheel build https://review.openstack.org/296130 | 01:01 |
| *** claudiub|2 has joined #openstack-infra | 01:02 | |
| *** sridhar_ram has quit IRC | 01:03 | |
| pabelanger | clarkb: ianw: Okay, calling it a night. Will do this tomorrow morning, hopefully jeblair will be around to answer questions too | 01:04 |
| *** claudiub|3 has joined #openstack-infra | 01:04 | |
| jeblair | clarkb, pabelanger: correct, volume-per-wheel-arch. basically, since the wheel-arch mirrors are built separately, we want to vos-release them separately, and the unit for doing that is a volume. | 01:04 |
| pabelanger | we are close | 01:04 |
| pabelanger | oh | 01:04 |
| jeblair | sorry, i'm around. no one said my name till now :) | 01:05 |
| pabelanger | np | 01:05 |
| jeblair | but i was about to leave, so happy to pick up tomorrow if you want :) | 01:05 |
| pabelanger | Ya, lets do that | 01:05 |
| pabelanger | I'll read up on the docs too | 01:05 |
| *** claudiub has quit IRC | 01:05 | |
| jeblair | yah, that'll be good. then i can fill in the blanks. always good to test the docs | 01:06 |
| jeblair | goodnight :) | 01:06 |
| pabelanger | yup, nite all | 01:06 |
| *** mylu has joined #openstack-infra | 01:06 | |
| *** claudiub|2 has quit IRC | 01:08 | |
| *** heyongli has quit IRC | 01:09 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:09 | |
| *** heyongli has joined #openstack-infra | 01:09 | |
| *** achanda has joined #openstack-infra | 01:11 | |
| *** Sukhdev has quit IRC | 01:12 | |
| *** Qiming has joined #openstack-infra | 01:13 | |
| fungi | mordred: i know you're afk, but good news is the modifications seem to have mostly worked. i think we still have a cache issue though, as it ended up reporting a previous try's uuid when it completed rather than the one it built | 01:15 |
| *** zhurong has joined #openstack-infra | 01:15 | |
| *** piet has quit IRC | 01:15 | |
| *** EricGonczer_ has quit IRC | 01:17 | |
| *** fawadkhaliq has quit IRC | 01:18 | |
| *** fawadkhaliq has joined #openstack-infra | 01:18 | |
| *** heyongli has quit IRC | 01:19 | |
| *** fawadkhaliq has quit IRC | 01:19 | |
| *** heyongli has joined #openstack-infra | 01:19 | |
| *** fawadkhaliq has joined #openstack-infra | 01:20 | |
| *** rfolco has joined #openstack-infra | 01:20 | |
| *** mylu has quit IRC | 01:23 | |
| *** mtanino has joined #openstack-infra | 01:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 01:24 |
| *** baoli has joined #openstack-infra | 01:24 | |
| *** tphummel has quit IRC | 01:25 | |
| *** tphummel has joined #openstack-infra | 01:25 | |
| *** thorst_ has quit IRC | 01:27 | |
| *** thorst has joined #openstack-infra | 01:27 | |
| *** sfinucan has quit IRC | 01:28 | |
| *** heyongli has quit IRC | 01:29 | |
| *** heyongli has joined #openstack-infra | 01:30 | |
| *** baoli has quit IRC | 01:30 | |
| *** sshnaidm has joined #openstack-infra | 01:30 | |
| *** mriedem has quit IRC | 01:31 | |
| *** ayoung has joined #openstack-infra | 01:32 | |
| *** mylu has joined #openstack-infra | 01:33 | |
| *** baoli has joined #openstack-infra | 01:34 | |
| *** salv-orlando has joined #openstack-infra | 01:35 | |
| *** ddieterly is now known as ddieterly[away] | 01:36 | |
| *** thorst has quit IRC | 01:36 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Further updates to defaulting to alt election repo https://review.openstack.org/296141 | 01:37 |
| *** zz_dimtruck is now known as dimtruck | 01:38 | |
| *** IlyaG has joined #openstack-infra | 01:38 | |
| *** doublek has quit IRC | 01:38 | |
| *** heyongli has quit IRC | 01:39 | |
| *** heyongli has joined #openstack-infra | 01:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Refactor the v3 only job https://review.openstack.org/271128 | 01:40 |
| *** dims_ has quit IRC | 01:40 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/puppet-httpd: Add support for alias and directory variables https://review.openstack.org/292674 | 01:40 |
| *** salv-orlando has quit IRC | 01:42 | |
| *** yamahata has quit IRC | 01:43 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Publish election data to static server https://review.openstack.org/292676 | 01:44 |
| *** Daisy has joined #openstack-infra | 01:46 | |
| *** EricGonczer_ has joined #openstack-infra | 01:47 | |
| *** heyongli has quit IRC | 01:50 | |
| *** heyongli has joined #openstack-infra | 01:50 | |
| *** cody-somerville has quit IRC | 01:52 | |
| *** mylu has quit IRC | 01:54 | |
| *** amitgandhinz has joined #openstack-infra | 01:56 | |
| *** mylu has joined #openstack-infra | 01:57 | |
| *** tpsilva has quit IRC | 01:57 | |
| *** mylu has quit IRC | 01:58 | |
| *** heyongli has quit IRC | 02:00 | |
| *** heyongli has joined #openstack-infra | 02:00 | |
| *** mylu has joined #openstack-infra | 02:00 | |
| *** fawadkhaliq has quit IRC | 02:01 | |
| *** fawadkhaliq has joined #openstack-infra | 02:02 | |
| *** ddieterly[away] is now known as ddieterly | 02:02 | |
| *** xiangxinyong has quit IRC | 02:03 | |
| *** xiangxinyong has joined #openstack-infra | 02:04 | |
| *** tonytan4ever has joined #openstack-infra | 02:06 | |
| *** verdurin has quit IRC | 02:07 | |
| *** cody-somerville has joined #openstack-infra | 02:08 | |
| *** rfolco has quit IRC | 02:08 | |
| *** lezbar__ has quit IRC | 02:08 | |
| *** tonytan_brb has joined #openstack-infra | 02:08 | |
| *** tonytan4ever has quit IRC | 02:09 | |
| *** lezbar has joined #openstack-infra | 02:09 | |
| *** amitgandhinz has quit IRC | 02:09 | |
| *** tonytan4ever has joined #openstack-infra | 02:09 | |
| *** fawadkhaliq has quit IRC | 02:10 | |
| *** verdurin has joined #openstack-infra | 02:10 | |
| *** fawadkhaliq has joined #openstack-infra | 02:10 | |
| *** heyongli has quit IRC | 02:10 | |
| *** heyongli has joined #openstack-infra | 02:11 | |
| *** bpokorny_ has joined #openstack-infra | 02:11 | |
| *** bpokorny_ has quit IRC | 02:11 | |
| *** bryan_att has quit IRC | 02:12 | |
| *** tonytan_brb has quit IRC | 02:13 | |
| *** ddieterly is now known as ddieterly[away] | 02:13 | |
| *** tphummel has quit IRC | 02:14 | |
| *** bpokorny has quit IRC | 02:15 | |
| *** mylu has quit IRC | 02:15 | |
| *** mylu has joined #openstack-infra | 02:17 | |
| *** claudiub has joined #openstack-infra | 02:17 | |
| openstackgerrit | Adam Spiers proposed openstack-infra/project-config: Create openstack-resource-agents-specs git repo https://review.openstack.org/296144 | 02:17 |
| *** kzaitsev_mb has quit IRC | 02:17 | |
| *** rkukura has quit IRC | 02:19 | |
| *** claudiub|3 has quit IRC | 02:20 | |
| *** heyongli has quit IRC | 02:20 | |
| *** heyongli has joined #openstack-infra | 02:21 | |
| *** EricGonczer_ has quit IRC | 02:21 | |
| *** IlyaG has quit IRC | 02:24 | |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Enable funcitonal check for zaqar client https://review.openstack.org/296147 | 02:25 |
| *** apoorvad has quit IRC | 02:26 | |
| *** woodster_ has quit IRC | 02:27 | |
| *** Daisy has quit IRC | 02:29 | |
| *** Daisy has joined #openstack-infra | 02:30 | |
| *** heyongli has quit IRC | 02:31 | |
| *** dimtruck is now known as zz_dimtruck | 02:31 | |
| *** heyongli has joined #openstack-infra | 02:31 | |
| *** amrith is now known as _amrith_ | 02:32 | |
| *** akshai_ has quit IRC | 02:32 | |
| *** hichihara has joined #openstack-infra | 02:33 | |
| *** thorst has joined #openstack-infra | 02:34 | |
| *** knikolla has quit IRC | 02:34 | |
| *** mylu has quit IRC | 02:38 | |
| *** mylu has joined #openstack-infra | 02:39 | |
| *** akshai has joined #openstack-infra | 02:40 | |
| *** MarkAtwood has joined #openstack-infra | 02:40 | |
| *** thorst has quit IRC | 02:41 | |
| *** heyongli has quit IRC | 02:41 | |
| *** salv-orlando has joined #openstack-infra | 02:41 | |
| *** heyongli has joined #openstack-infra | 02:41 | |
| *** salv-orlando has quit IRC | 02:51 | |
| *** heyongli has quit IRC | 02:51 | |
| *** fawadkhaliq has quit IRC | 02:51 | |
| *** heyongli has joined #openstack-infra | 02:51 | |
| *** fawadkhaliq has joined #openstack-infra | 02:52 | |
| *** markvoelker has quit IRC | 02:59 | |
| tonyb | AJaeger: I don't know if you will have seen it but a bunch of the periodic-stable jobs filed last night like: http://lists.openstack.org/pipermail/openstack-stable-maint/2016-March/004155.html (basically the *-db* jobs came back NOT_REGISTERED) | 03:01 |
| *** fawadkhaliq has quit IRC | 03:01 | |
| *** fawadkhaliq has joined #openstack-infra | 03:01 | |
| *** fawadkhaliq has quit IRC | 03:01 | |
| *** heyongli has quit IRC | 03:01 | |
| *** fawadkhaliq has joined #openstack-infra | 03:02 | |
| *** heyongli has joined #openstack-infra | 03:02 | |
| fungi | okay, i'm wanting to locally test some modified puppet modules on a replacement server i spun up... aside from making sure i disable it by uuid on the master, what do i need to do to get it to use a modified module source instead of the real one? now that we're not having servers contact a puppetmaster i assume i can't just make an environment there and get it to use that any longer | 03:05 |
| *** amitgandhinz has joined #openstack-infra | 03:06 | |
| jpmaxman | fungi: just wanted to check on the new OpenStackID server. Is that up and running or do you have an idea of when it would be? We don't want to submit to the app store until it's solid. | 03:07 |
| fungi | jpmaxman: i've been talking with smarcet about it elsewhere because he's not in irc | 03:07 |
| jpmaxman | Sorry if others have saved just getting caught up :) | 03:07 |
| jpmaxman | Ahh ok | 03:08 |
| fungi | we're trying to work out whether laravel should be getting the fqdn or short hostname passed in | 03:08 |
| jpmaxman | Ok so a wip still :) got it. | 03:09 |
| fungi | apparently we deployed our old servers with the fqdn in /etc/hostname but in more recent years we've started deploying it with the short hostname in /etc/hostname and relying on hostname -f to return the fqdn | 03:09 |
| *** piet has joined #openstack-infra | 03:10 | |
| jpmaxman | I see and that's throwing off the laravel app I guess | 03:10 |
| fungi | which is technically more correct, but means software which was relying on it being one way is now going to break if we try to make it expect the other | 03:10 |
| jpmaxman | Seems like an easy change OTOH with every change... | 03:11 |
| jpmaxman | Ok thanks for the update | 03:11 |
| openstackgerrit | Adam Spiers proposed openstack-infra/project-config: Create openstack-resource-agents-specs git repo https://review.openstack.org/296144 | 03:11 |
| fungi | jpmaxman: well, if we switch it so that the replacement server is happy, the current production server will probably go to lunch | 03:11 |
| fungi | but it's also very late here and i'm ceasing to think particularly straight now | 03:12 |
| *** heyongli has quit IRC | 03:12 | |
| *** heyongli has joined #openstack-infra | 03:12 | |
| jpmaxman | I am really wondering if we shouldn't keep the server we have and get through the app approval then look at upgrading it. | 03:12 |
| fungi | it may be easiest to manually modify /etc/hostname because i'm pretty sure puppet isn't touching that after the server is built | 03:12 |
| fungi | otherwise the current server would have been changed to match long ago | 03:13 |
| jpmaxman | Hmm. Would rather get it right. | 03:13 |
| *** ddieterly[away] has quit IRC | 03:14 | |
| jpmaxman | We are going to quickly have to work on scalability and redundancy for this service. Having it buttoned up would be necessary toward that end. | 03:14 |
| *** kzaitsev_mb has joined #openstack-infra | 03:15 | |
| *** Qiming has quit IRC | 03:15 | |
| jpmaxman | Ok let me discuss with smarcet and Jimmy go see what their take is as well. What's your personal preference (if you have one)? | 03:15 |
| fungi | i agree. probably adjusting the puppet module to pass $::hostname instead of $::fqdn into the laravel config template and simultaneously modifying the current production server's /etc/hostname to the short hostname would solve this, and make future redeployments on new servers just work | 03:17 |
| *** mylu has quit IRC | 03:17 | |
| *** claudiub has quit IRC | 03:17 | |
| *** amitgandhinz has quit IRC | 03:19 | |
| *** yuanying has quit IRC | 03:21 | |
| *** yuanying_ has joined #openstack-infra | 03:21 | |
| *** heyongli has quit IRC | 03:22 | |
| *** mylu has joined #openstack-infra | 03:22 | |
| xiangxinyong | Hi guys, if you have time, please take a look at it. https://bugs.launchpad.net/hacking/+bug/1560775 | 03:22 |
| openstack | Launchpad bug 1560775 in hacking "H903 is not working" [Undecided,New] | 03:22 |
| *** heyongli has joined #openstack-infra | 03:22 | |
| *** IlyaG has joined #openstack-infra | 03:22 | |
| *** piet has quit IRC | 03:23 | |
| *** yuanying_ has quit IRC | 03:24 | |
| *** yuanying has joined #openstack-infra | 03:24 | |
| fungi | well, changing the hostname of the replacement server to be the fqdn so it's like the current production server doesn't seem to be making any difference | 03:25 |
| *** yamahata has joined #openstack-infra | 03:26 | |
| *** asettle has quit IRC | 03:28 | |
| *** links has joined #openstack-infra | 03:28 | |
| *** yuanying has quit IRC | 03:29 | |
| *** jerryz has quit IRC | 03:30 | |
| *** heyongli has quit IRC | 03:32 | |
| *** heyongli has joined #openstack-infra | 03:33 | |
| jpmaxman | Ok thanks for the input. | 03:35 |
| jpmaxman | That makes sense to me. I haven't heard back from smarcet. But I am about to call it a night. We can pick up in the morning. I really appreciate all your help with this :) | 03:36 |
| *** thorst has joined #openstack-infra | 03:39 | |
| *** hamzy has joined #openstack-infra | 03:40 | |
| *** jamesmcarthur has joined #openstack-infra | 03:41 | |
| fungi | jpmaxman: yeah, this may be easier to troubleshoot tomorrow when it's not rapidly approaching midnight for me | 03:41 |
| fungi | i think smarcet has already dropped off for the night anyway | 03:41 |
| *** IlyaG has quit IRC | 03:42 | |
| *** heyongli has quit IRC | 03:42 | |
| *** Daisy has quit IRC | 03:43 | |
| *** heyongli has joined #openstack-infra | 03:43 | |
| *** Daisy has joined #openstack-infra | 03:43 | |
| *** thorst has quit IRC | 03:46 | |
| *** baoli has quit IRC | 03:46 | |
| *** jamesmcarthur has quit IRC | 03:47 | |
| *** Daisy has quit IRC | 03:47 | |
| *** jamesmcarthur has joined #openstack-infra | 03:51 | |
| *** heyongli has quit IRC | 03:53 | |
| *** heyongli has joined #openstack-infra | 03:53 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** kushal has quit IRC | 03:57 | |
| *** markvoelker has joined #openstack-infra | 03:59 | |
| *** yuanying has joined #openstack-infra | 04:01 | |
| *** jamesmcarthur has joined #openstack-infra | 04:02 | |
| *** heyongli has quit IRC | 04:03 | |
| *** heyongli has joined #openstack-infra | 04:03 | |
| *** markvoelker has quit IRC | 04:04 | |
| *** cody-somerville has quit IRC | 04:05 | |
| *** achanda has quit IRC | 04:06 | |
| *** asettle has joined #openstack-infra | 04:06 | |
| *** salv-orlando has joined #openstack-infra | 04:10 | |
| *** asettle has quit IRC | 04:10 | |
| *** tonytan4ever has quit IRC | 04:11 | |
| *** baoli has joined #openstack-infra | 04:13 | |
| *** heyongli has quit IRC | 04:13 | |
| *** heyongli has joined #openstack-infra | 04:14 | |
| *** salv-orlando has quit IRC | 04:14 | |
| *** baoli has quit IRC | 04:14 | |
| *** baoli has joined #openstack-infra | 04:15 | |
| *** amitgandhinz has joined #openstack-infra | 04:16 | |
| *** fawadkhaliq has quit IRC | 04:17 | |
| *** Nikolay_St has quit IRC | 04:17 | |
| *** fawadkhaliq has joined #openstack-infra | 04:17 | |
| *** jamesmcarthur has quit IRC | 04:18 | |
| *** heyongli has quit IRC | 04:23 | |
| *** baoli has quit IRC | 04:24 | |
| *** heyongli has joined #openstack-infra | 04:24 | |
| *** treaki_ has joined #openstack-infra | 04:24 | |
| *** treaki__ has quit IRC | 04:26 | |
| *** kushal has joined #openstack-infra | 04:27 | |
| *** jamesmcarthur has joined #openstack-infra | 04:28 | |
| *** amitgandhinz has quit IRC | 04:29 | |
| *** nandhu has quit IRC | 04:30 | |
| *** yamamoto_ has joined #openstack-infra | 04:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Make reproduce function quiet https://review.openstack.org/296171 | 04:31 |
| *** mylu has quit IRC | 04:32 | |
| *** heyongli has quit IRC | 04:34 | |
| *** heyongli has joined #openstack-infra | 04:34 | |
| *** fawadkhaliq has quit IRC | 04:35 | |
| *** fawadkhaliq has joined #openstack-infra | 04:36 | |
| *** mylu has joined #openstack-infra | 04:36 | |
| *** mylu has quit IRC | 04:38 | |
| *** beisner- has joined #openstack-infra | 04:38 | |
| *** beisner has quit IRC | 04:39 | |
| *** beisner- is now known as beisner | 04:39 | |
| *** gaurangt has joined #openstack-infra | 04:39 | |
| *** fawadkhaliq has quit IRC | 04:40 | |
| *** fawadkhaliq has joined #openstack-infra | 04:41 | |
| *** fawadkhaliq has quit IRC | 04:41 | |
| *** fawadkhaliq has joined #openstack-infra | 04:42 | |
| *** fawadkhaliq has quit IRC | 04:42 | |
| *** jamesmcarthur has quit IRC | 04:42 | |
| *** IlyaG has joined #openstack-infra | 04:42 | |
| *** fawadkhaliq has joined #openstack-infra | 04:43 | |
| *** thorst has joined #openstack-infra | 04:44 | |
| *** heyongli has quit IRC | 04:44 | |
| *** heyongli has joined #openstack-infra | 04:44 | |
| *** fawadkhaliq has quit IRC | 04:46 | |
| *** fawadkhaliq has joined #openstack-infra | 04:46 | |
| *** fawadkhaliq has quit IRC | 04:47 | |
| *** IlyaG has quit IRC | 04:47 | |
| *** fawadkhaliq has joined #openstack-infra | 04:48 | |
| *** fawadkhaliq has quit IRC | 04:48 | |
| *** fawadkhaliq has joined #openstack-infra | 04:49 | |
| *** mylu has joined #openstack-infra | 04:50 | |
| *** fedexo has joined #openstack-infra | 04:51 | |
| *** thorst has quit IRC | 04:52 | |
| *** achanda has joined #openstack-infra | 04:53 | |
| *** heyongli has quit IRC | 04:54 | |
| *** heyongli has joined #openstack-infra | 04:55 | |
| ianw | pabelanger: http://paste.openstack.org/show/491525/ | 04:56 |
| ianw | pabelanger: so, the packages that are in this weird "need to be updated but already installed" state are the ones that have changed since the last run (good.txt v bad.txt). so this points to something in the way we're handling the cache between runs | 04:58 |
| ianw | pabelanger: exactly what that is ... i don't quite know yet. but getting closer | 04:58 |
| *** MarkAtwood has quit IRC | 05:04 | |
| *** heyongli has quit IRC | 05:04 | |
| lifeless | ianw: rpm cache? | 05:05 |
| *** heyongli has joined #openstack-infra | 05:05 | |
| *** heyongli has quit IRC | 05:05 | |
| *** akshai has quit IRC | 05:07 | |
| ianw | lifeless: yes, debugging dib failures | 05:09 |
| *** gaurangt1 has joined #openstack-infra | 05:10 | |
| *** gaurangt has quit IRC | 05:11 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Fix test for new WebOb https://review.openstack.org/296180 | 05:12 |
| *** tonytan4ever has joined #openstack-infra | 05:13 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 05:14 |
| *** Daisy has joined #openstack-infra | 05:15 | |
| *** sridhar_ram has joined #openstack-infra | 05:16 | |
| *** mylu has quit IRC | 05:17 | |
| *** tonytan4ever has quit IRC | 05:17 | |
| *** gaurangt1 has left #openstack-infra | 05:18 | |
| *** Qiming has joined #openstack-infra | 05:18 | |
| *** Daisy has quit IRC | 05:19 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Fix test for new WebOb https://review.openstack.org/296180 | 05:23 |
| *** amitgandhinz has joined #openstack-infra | 05:26 | |
| Nakato | So, the gate for project openstack-infra/zuul is bust from an upstream project release. I've patched up the test: ^ | 05:35 |
| *** kzaitsev_mb has quit IRC | 05:35 | |
| *** MarkAtwood has joined #openstack-infra | 05:36 | |
| *** amitgandhinz has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack-infra/project-config: trove: Use db setup for python jobs https://review.openstack.org/295046 | 05:39 |
| *** salv-orlando has joined #openstack-infra | 05:42 | |
| *** roxanaghe has joined #openstack-infra | 05:43 | |
| openstackgerrit | Merged openstack-infra/project-config: sqlalchemy-migrate: Use db setup for python jobs https://review.openstack.org/295435 | 05:43 |
| AJaeger | tonyb: yes, seen it - those run at the wrong time, we have a small window where this can happen when zuul is setup first but not jenkins. It should be fine today, if not I'll investigate. | 05:45 |
| AJaeger | ianw: could you +2A https://review.openstack.org/#/c/295035 , please? | 05:46 |
| *** salv-orlando has quit IRC | 05:49 | |
| *** thorst has joined #openstack-infra | 05:49 | |
| *** roxanaghe has quit IRC | 05:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Support py34 functional tests for Oslo.Messaging https://review.openstack.org/295531 | 05:51 |
| *** Daisy has joined #openstack-infra | 05:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch voting to false for oslo.messaging+dsvm jobs https://review.openstack.org/296093 | 05:53 |
| *** Nikolay_St has joined #openstack-infra | 05:53 | |
| *** asettle has joined #openstack-infra | 05:55 | |
| *** thorst has quit IRC | 05:55 | |
| tonyb | AJaeger: cool. Thanks. | 05:57 |
| *** Sukhdev has joined #openstack-infra | 05:57 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Really remove all interfaces in dhcp-all-ifaces https://review.openstack.org/296186 | 05:58 |
| tonyb | Could somethign have changed in the noepool images in the last 5 days that would alter the locale settings? | 05:59 |
| *** Qiming_ has joined #openstack-infra | 06:00 | |
| *** markvoelker has joined #openstack-infra | 06:00 | |
| tonyb | I'm seeing https://bugs.launchpad.net/openstack-gate/+bug/1560808 a debug job I ran says the default encoding is "ANSI_X3.4-1968" which seems like an odd choice. | 06:01 |
| openstack | Launchpad bug 1560808 in OpenStack-Gate "stable/liberty: gate-requirements-integration-dsvm fails with due to UnicodeDecodeError in httpretty" [Undecided,New] | 06:01 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Handle unconfigured interfaces for dhcp-all-ifaces https://review.openstack.org/296187 | 06:02 |
| *** Qiming has quit IRC | 06:02 | |
| *** abregman has joined #openstack-infra | 06:03 | |
| *** fawadkhaliq has quit IRC | 06:03 | |
| *** fawadkhaliq has joined #openstack-infra | 06:04 | |
| *** markvoelker has quit IRC | 06:04 | |
| *** camunoz has quit IRC | 06:07 | |
| AJaeger | tonyb: we switched to ubuntu-trusty images and those had different locale - but that's a bit longer than 5 days | 06:08 |
| *** abregman has quit IRC | 06:08 | |
| tonyb | AJaeger: well that could be it, IIRC all thre passes are devstack-trusty and the fails are ubuntu-trusty. | 06:09 |
| tonyb | it is howveer strange that it only seesm to trigger on stable/liberty (not kilo or mitaka/master) | 06:09 |
| *** abregman has joined #openstack-infra | 06:10 | |
| *** SumitNaiksatam has quit IRC | 06:10 | |
| tonyb | TBH I'm grasping at straws, but I can't reproduce it outside of the gate. Tomorrow I'll learn enough DIB to build somethign lose to the nodepoll image for testing | 06:10 |
| tonyb | (s/lose/close/) | 06:10 |
| greghaynes | disk-image-create ubuntu-minimal vm | 06:10 |
| greghaynes | :) | 06:11 |
| tonyb | greghaynes: nothign is that simple ;P | 06:11 |
| AJaeger | tonyb: thta would explain it. We made some changes for default locale but I'm not sure whether all are in. | 06:11 |
| greghaynes | append a "simple-init" if you want to upload it to a cloud | 06:11 |
| greghaynes | tonyb: yea, thats true | 06:11 |
| greghaynes | I didnt say it would *work* | 06:11 |
| tonyb | greghaynes: touche | 06:12 |
| AJaeger | tonyb: there's still https://review.openstack.org/#/c/290789/ by yolanda, not sure whether that is needed. Best talk to her later... | 06:12 |
| AJaeger | back later... | 06:12 |
| *** achanda_ has joined #openstack-infra | 06:13 | |
| tonyb | AJaeger: thanks I'll try with that tomorrow. | 06:13 |
| *** achanda has quit IRC | 06:14 | |
| tonyb | greghaynes: given that you just +2A'd that will it be used to generate nodes tonight? | 06:17 |
| greghaynes | tonyb: yep | 06:18 |
| tonyb | greghaynes: \o/ | 06:18 |
| greghaynes | tonyb: Something that concerns me though - I dont see where we install the language-pack-en package | 06:18 |
| greghaynes | I might just not be able to find it, but we'll see if thats an issue | 06:18 |
| *** cody-somerville has joined #openstack-infra | 06:19 | |
| tonyb | greghaynes: Oh. | 06:19 |
| greghaynes | regardless though, we should be setting the locale | 06:19 |
| *** fawadkhaliq has quit IRC | 06:20 | |
| tonyb | greghaynes: Yeah the default seems strange. | 06:20 |
| *** fawadkhaliq has joined #openstack-infra | 06:21 | |
| *** ihrachys has joined #openstack-infra | 06:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/296197 | 06:23 |
| *** IlyaG has joined #openstack-infra | 06:23 | |
| tonyb | greghaynes: huzzah! setting my local to match the node images and I can reproduce the failure so #winning | 06:23 |
| greghaynes | tonyb: nice | 06:23 |
| *** asettle has quit IRC | 06:25 | |
| tonyb | and with that I think I'll make the family dinner. | 06:25 |
| tonyb | greghaynes, AJaeger: thanks! | 06:25 |
| *** fawadkhaliq has quit IRC | 06:25 | |
| *** fawadkhaliq has joined #openstack-infra | 06:25 | |
| *** fawadkhaliq has quit IRC | 06:26 | |
| *** fawadkhaliq has joined #openstack-infra | 06:26 | |
| *** fawadkhaliq has quit IRC | 06:27 | |
| *** fawadkhaliq has joined #openstack-infra | 06:28 | |
| *** IlyaG has quit IRC | 06:28 | |
| *** fawadkhaliq has quit IRC | 06:28 | |
| *** nadya has joined #openstack-infra | 06:29 | |
| *** fawadkhaliq has joined #openstack-infra | 06:29 | |
| *** ihrachys has quit IRC | 06:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:32 | |
| *** Daisy has quit IRC | 06:33 | |
| *** Daisy has joined #openstack-infra | 06:33 | |
| *** fedexo has quit IRC | 06:35 | |
| *** sdake_ has joined #openstack-infra | 06:35 | |
| *** amitgandhinz has joined #openstack-infra | 06:35 | |
| *** sdake has quit IRC | 06:36 | |
| openstackgerrit | Koteswara Rao Kelam proposed openstack-infra/project-config: Adding networking-hpe project. https://review.openstack.org/295775 | 06:36 |
| *** kzaitsev_mb has quit IRC | 06:37 | |
| prometheanfire | greghaynes: oh hi :P | 06:37 |
| prometheanfire | if you aren't busy https://review.openstack.org/#/c/294973/ | 06:38 |
| *** Daisy has quit IRC | 06:38 | |
| *** nikil89_ has joined #openstack-infra | 06:41 | |
| *** nikil89_ is now known as naggappan | 06:41 | |
| *** sdake_ has quit IRC | 06:44 | |
| *** Daisy has joined #openstack-infra | 06:44 | |
| *** sdake has joined #openstack-infra | 06:44 | |
| *** jcooley_ has quit IRC | 06:45 | |
| *** nadya has quit IRC | 06:45 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:45 |
| *** sridhar_ram has quit IRC | 06:46 | |
| *** MarkAtwood has quit IRC | 06:47 | |
| *** amitgandhinz has quit IRC | 06:49 | |
| openstackgerrit | Merged openstack/diskimage-builder: Set default locale to image in debootstrap element https://review.openstack.org/290789 | 06:52 |
| *** vgridnev has joined #openstack-infra | 06:53 | |
| *** thorst has joined #openstack-infra | 06:53 | |
| abregman | morning | 06:55 |
| *** salv-orlando has joined #openstack-infra | 06:58 | |
| *** e0ne has joined #openstack-infra | 06:58 | |
| *** thorst has quit IRC | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** gildub has quit IRC | 07:02 | |
| *** watanabe_isao has joined #openstack-infra | 07:06 | |
| *** markvoelker has quit IRC | 07:07 | |
| *** vikram has joined #openstack-infra | 07:14 | |
| openstackgerrit | Ivan Chavero proposed openstack-infra/project-config: Add tarball creation to packstack project https://review.openstack.org/296084 | 07:15 |
| *** salv-orlando has quit IRC | 07:15 | |
| *** asettle has joined #openstack-infra | 07:18 | |
| *** sridhar_ram has joined #openstack-infra | 07:19 | |
| *** sridhar_ram has quit IRC | 07:19 | |
| *** sdake has quit IRC | 07:23 | |
| *** asettle has quit IRC | 07:23 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Log shade inner exceptions https://review.openstack.org/225361 | 07:24 |
| yolanda | good morning | 07:24 |
| *** Sukhdev has quit IRC | 07:25 | |
| *** davidlenwell has quit IRC | 07:26 | |
| *** achanda_ has quit IRC | 07:26 | |
| AJaeger | good morning, yolanda ! | 07:27 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/296197 | 07:28 |
| AJaeger | yolanda: https://review.openstack.org/#/c/295035 is ready to merge (db change), so a +2A would be great | 07:28 |
| *** e0ne has quit IRC | 07:28 | |
| AJaeger | And then later a review of https://review.openstack.org/295044 and https://review.openstack.org/294949, plesae - but those need a second +2... | 07:29 |
| yolanda | ok review time! | 07:30 |
| AJaeger | thanks, yolanda ! | 07:31 |
| *** akaszuba has joined #openstack-infra | 07:32 | |
| yolanda | mm, i'm seeing an unstable job in jenkins, problems copying files | 07:32 |
| yolanda | i wonder if we are out of disk again | 07:32 |
| vikram | AJaeger: I would like to add neutron sub-project for one of the huawei's in-house controller called 'AC'.. The scope will be using Neutron's API and making the interaction between Neutron and AC possible.. Do I need to follow "http://docs.openstack.org/developer/neutron/devref/sub_projects.html"? | 07:32 |
| *** roxanaghe has joined #openstack-infra | 07:32 | |
| *** mikelk has joined #openstack-infra | 07:33 | |
| yolanda | mm, no, we are fine. Permission denied errors, so that's something related to the job | 07:33 |
| *** kzaitsev_mb has joined #openstack-infra | 07:33 | |
| *** davidlenwell has joined #openstack-infra | 07:34 | |
| AJaeger | https://review.openstack.org/#/c/295036/ is ready to go... | 07:34 |
| AJaeger | yolanda: Ah, the new jenkins fails on permission denied - the old ignored them | 07:34 |
| AJaeger | vikram: talk with armax first whether this will be part of neutron or not. | 07:34 |
| vikram | AJaeger: ok.. | 07:35 |
| AJaeger | vikram: in any case, the Infra Manual will give all infra related changes, see http://docs.openstack.org/infra/manual/creators.html | 07:35 |
| *** achanda has joined #openstack-infra | 07:35 | |
| vikram | AJaeger: If not in neutron then probably we got it to be a stackforge project? | 07:35 |
| AJaeger | vikram: we don't have the name stackforge anymore, just the concept of non-official projects. REad the Creator's guide, it gives details on both. | 07:36 |
| *** roxanaghe has quit IRC | 07:36 | |
| vikram | AJaeger: Got it.. | 07:37 |
| *** kzaitsev_mb has quit IRC | 07:38 | |
| *** nmagnezi has joined #openstack-infra | 07:39 | |
| openstackgerrit | Merged openstack-infra/project-config: rack,sahara, tacker: Use db setup for python jobs https://review.openstack.org/295035 | 07:39 |
| *** tesseract has joined #openstack-infra | 07:41 | |
| *** tesseract is now known as Guest60842 | 07:41 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 07:42 |
| *** amitgandhinz has joined #openstack-infra | 07:45 | |
| *** dizquierdo has joined #openstack-infra | 07:46 | |
| *** vgridnev has quit IRC | 07:49 | |
| *** asettle has joined #openstack-infra | 07:51 | |
| *** zeih has joined #openstack-infra | 07:55 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 07:57 |
| *** vgridnev has joined #openstack-infra | 07:58 | |
| *** thorst has joined #openstack-infra | 07:58 | |
| *** amitgandhinz has quit IRC | 07:58 | |
| *** dtardivel has joined #openstack-infra | 08:00 | |
| *** Daisy has quit IRC | 08:02 | |
| *** Daisy has joined #openstack-infra | 08:02 | |
| *** Nikolay_St has quit IRC | 08:04 | |
| *** thorst has quit IRC | 08:06 | |
| *** arxcruz has joined #openstack-infra | 08:06 | |
| *** achanda has quit IRC | 08:06 | |
| *** jaosorior has joined #openstack-infra | 08:06 | |
| *** Daisy has quit IRC | 08:07 | |
| *** achanda has joined #openstack-infra | 08:08 | |
| *** sflanigan has quit IRC | 08:10 | |
| *** achanda has quit IRC | 08:11 | |
| *** achanda has joined #openstack-infra | 08:12 | |
| *** claudiub has joined #openstack-infra | 08:12 | |
| *** achanda has quit IRC | 08:12 | |
| *** achanda has joined #openstack-infra | 08:13 | |
| *** matrohon has joined #openstack-infra | 08:13 | |
| *** achanda has quit IRC | 08:13 | |
| AJaeger | yolanda: did we looose openstackgerrit ? | 08:13 |
| yolanda | going to check | 08:13 |
| AJaeger | yolanda: wait | 08:14 |
| AJaeger | The changes did not merge, I was confused... | 08:14 |
| yolanda | ok :) | 08:14 |
| openstackgerrit | Merged openstack-infra/project-config: Move lbaas v2 driver choice into gate hook https://review.openstack.org/295488 | 08:14 |
| yolanda | works | 08:15 |
| openstackgerrit | Merged openstack-infra/project-config: add Synergy project https://review.openstack.org/283157 | 08:15 |
| openstackgerrit | Merged openstack-infra/project-config: Add NFS devstack plugin https://review.openstack.org/289589 | 08:15 |
| *** vgridnev has quit IRC | 08:15 | |
| *** zhurong has quit IRC | 08:16 | |
| AJaeger | yolanda: indeed ;) | 08:17 |
| *** _skolekonov is now known as skolekonov | 08:19 | |
| *** chem has quit IRC | 08:22 | |
| *** dizquierdo has quit IRC | 08:23 | |
| *** vincentll has joined #openstack-infra | 08:24 | |
| AJaeger | yolanda: this one is ready as well, if you have time for review later: https://review.openstack.org/#/c/293769/ | 08:25 |
| *** MarkAtwood has joined #openstack-infra | 08:26 | |
| *** chem has joined #openstack-infra | 08:26 | |
| openstackgerrit | Merged openstack-infra/system-config: Don't pass OS_CLOUD and OS_REGION_NAME to expand-groups https://review.openstack.org/296127 | 08:26 |
| *** Daisy has joined #openstack-infra | 08:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable funcitonal check for zaqar client https://review.openstack.org/296147 | 08:30 |
| openstackgerrit | Merged openstack-infra/infra-ansible: Fix tiny styling issue https://review.openstack.org/294781 | 08:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add tarball creation to packstack project https://review.openstack.org/296084 | 08:30 |
| *** ihrachys has joined #openstack-infra | 08:30 | |
| *** MarkAtwood has quit IRC | 08:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Creation of a dedicated repo for API and media storage client https://review.openstack.org/255349 | 08:31 |
| openstackgerrit | Merged openstack-infra/project-config: Don't integration test Packstack on release notes or docs patches https://review.openstack.org/295854 | 08:32 |
| *** roxanaghe has joined #openstack-infra | 08:33 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack-infra/project-config: Adding Mellanox Extension for the OpenStack Dashboard (Horizon) https://review.openstack.org/295260 | 08:34 |
| evgenyl | Hi, is there a way to install system packages before python tests are run? | 08:34 |
| *** zhurong has joined #openstack-infra | 08:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:34 | |
| *** esikachev has joined #openstack-infra | 08:35 | |
| AJaeger | evgenyl: what exactly do you want? | 08:35 |
| *** asettle has quit IRC | 08:37 | |
| *** roxanaghe has quit IRC | 08:37 | |
| *** shardy has joined #openstack-infra | 08:37 | |
| evgenyl | AJaeger: I have scipy as a dependency for my project, it requires lapack (linear algebra library) package to be installed, so when tests are run, it fails because cannot correctly install dependencies. | 08:37 |
| *** kzaitsev_mb has quit IRC | 08:39 | |
| *** watanabe_isao has quit IRC | 08:39 | |
| *** hichihara has quit IRC | 08:39 | |
| AJaeger | evgenyl: other-requirements.txt is the solution for this and we're preparing everything for this right now. | 08:40 |
| AJaeger | Expect an announcement soonish - but first we need to do a few more changes. | 08:40 |
| AJaeger | IF you want to try it as an experiment, see what fungi did here: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bindep | 08:41 |
| *** asettle has joined #openstack-infra | 08:41 | |
| openstackgerrit | Thomas Herve proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config https://review.openstack.org/296061 | 08:42 |
| AJaeger | You need to add all the packages that are in project-config/jenkins/data/bindep-fallback.txt also in your list - and we will do some cleanup of that list soonish. | 08:42 |
| *** permalac has joined #openstack-infra | 08:43 | |
| yolanda | i'm going to loose the connection for a pair of hours, internet improvements at home | 08:43 |
| AJaeger | evgenyl: if you want to experiment, please share the results with pabelanger, fungi, and myself. | 08:43 |
| *** kzaitsev_mb has joined #openstack-infra | 08:43 | |
| AJaeger | yolanda: hope it will be without problems ;) | 08:44 |
| evgenyl | AJaeger: that is awesome, looking forward to using it, thank you for the information. | 08:44 |
| *** Hal has joined #openstack-infra | 08:44 | |
| *** jed56 has joined #openstack-infra | 08:44 | |
| *** Hal is now known as Guest72269 | 08:45 | |
| *** jlanoux has joined #openstack-infra | 08:45 | |
| *** HeOS has joined #openstack-infra | 08:46 | |
| *** scheuran has joined #openstack-infra | 08:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental debian-jessie job for bindep-fallback https://review.openstack.org/293769 | 08:49 |
| *** dingyichen has quit IRC | 08:52 | |
| *** oanson has joined #openstack-infra | 08:55 | |
| *** amitgandhinz has joined #openstack-infra | 08:55 | |
| *** ramishra has quit IRC | 08:57 | |
| *** dizquierdo has joined #openstack-infra | 08:57 | |
| *** russellb has quit IRC | 08:58 | |
| *** ramishra has joined #openstack-infra | 08:59 | |
| *** russellb has joined #openstack-infra | 09:00 | |
| *** links has quit IRC | 09:01 | |
| *** piet has joined #openstack-infra | 09:01 | |
| openstackgerrit | Oleksandr Berezovskyi proposed openstack-infra/project-config: Add bareon-image-elements project https://review.openstack.org/295167 | 09:02 |
| *** e0ne has joined #openstack-infra | 09:03 | |
| *** rossella_s has quit IRC | 09:03 | |
| *** markvoelker has joined #openstack-infra | 09:03 | |
| *** thorst has joined #openstack-infra | 09:03 | |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** openstackgerrit has joined #openstack-infra | 09:04 | |
| *** rossella_s has joined #openstack-infra | 09:04 | |
| *** ramishra has quit IRC | 09:04 | |
| *** nmagnezi has quit IRC | 09:04 | |
| *** doug-fish has quit IRC | 09:04 | |
| *** salv-orlando has joined #openstack-infra | 09:06 | |
| *** claudiub|2 has joined #openstack-infra | 09:06 | |
| *** markvoelker has quit IRC | 09:08 | |
| *** vgridnev has joined #openstack-infra | 09:09 | |
| *** zhurong has quit IRC | 09:09 | |
| *** amitgandhinz has quit IRC | 09:09 | |
| *** ramishra has joined #openstack-infra | 09:09 | |
| *** claudiub has quit IRC | 09:09 | |
| *** thorst has quit IRC | 09:11 | |
| *** lucas-dinner is now known as lucasagomes | 09:13 | |
| *** zeih_ has joined #openstack-infra | 09:13 | |
| *** zeih has quit IRC | 09:13 | |
| *** achanda has joined #openstack-infra | 09:14 | |
| mattt | hi all, still struggling with an issue i can't quite figure out | 09:14 |
| *** Douhet has quit IRC | 09:14 | |
| mattt | we run ansible via tox in a gate job, and i cannot replicate the experience locally | 09:14 |
| *** sfinucan has joined #openstack-infra | 09:14 | |
| mattt | for example, we do a standard pip install in an ansible role and in the gate that pip install happens via /home/jenkins/workspace/gate-openstack-ansible-os_cinder-ansible-func-ubuntu-trusty/.tox/functional/bin/pip install lxc-python2 | 09:14 |
| *** Douhet has joined #openstack-infra | 09:15 | |
| *** piet has quit IRC | 09:16 | |
| mattt | i've already configured ansible to use ansible_python_interpreter=/home/jenkins/workspace/gate-openstack-ansible-os_cinder-ansible-func-ubuntu-trusty/.tox/functional/bin/python, but that does not result in my pip install tasks to use the tox pip binary like it does in the gate | 09:17 |
| *** ifarkas has joined #openstack-infra | 09:17 | |
| *** yolanda has quit IRC | 09:17 | |
| mattt | when i run locally, it always uses /usr/local/bin/pip | 09:17 |
| *** links has joined #openstack-infra | 09:18 | |
| *** derekh has joined #openstack-infra | 09:19 | |
| *** achanda has quit IRC | 09:20 | |
| *** nmagnezi has joined #openstack-infra | 09:21 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to reset backport-potential tags once backports are merged https://review.openstack.org/295993 | 09:21 |
| *** nelsnels_ has quit IRC | 09:22 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: ssh less https://review.openstack.org/295973 | 09:23 |
| *** kzaitsev_mb has quit IRC | 09:23 | |
| *** jlanoux has quit IRC | 09:24 | |
| *** yamahata has quit IRC | 09:27 | |
| *** jlanoux has joined #openstack-infra | 09:27 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: ssh less https://review.openstack.org/295973 | 09:29 |
| AJaeger | aspiers: you misunderstood me ;) time to talk about your specs repo? | 09:31 |
| aspiers | AJaeger: on IRC, yes. I'm in scrum standup :) | 09:31 |
| AJaeger | aspiers: let me just comment and you can answer once you can sit again ;) | 09:32 |
| AJaeger | aspiers: I'm talking about jobs that build your specs. Have a look at https://review.openstack.org/#/c/278777/ | 09:32 |
| aspiers | AJaeger: yes, I know | 09:33 |
| AJaeger | The gate job includes a doc job that builds the specs content and you can preview it, see http://docs-draft.openstack.org/77/278777/6/check/gate-infra-specs-docs/d8c863f//doc/build/html/ | 09:33 |
| *** esikachev has quit IRC | 09:33 | |
| *** jswarren has quit IRC | 09:33 | |
| AJaeger | So, that gives a way to check that your RST is valid. | 09:33 |
| aspiers | yes, I know | 09:33 |
| *** jswarren has joined #openstack-infra | 09:33 | |
| aspiers | but I don't have anywhere to build them to | 09:33 |
| aspiers | no -docs repo | 09:33 |
| *** yolanda has joined #openstack-infra | 09:34 | |
| aspiers | that's why patch-set 1 failed | 09:34 |
| *** roxanaghe has joined #openstack-infra | 09:34 | |
| AJaeger | aspiers: you do not need a docs repo. | 09:34 |
| AJaeger | You just cannot *publish* it to specs.openstack.org | 09:34 |
| aspiers | I know that too :) | 09:34 |
| aspiers | I read the comment in the templates | 09:34 |
| *** daemontool has joined #openstack-infra | 09:35 | |
| AJaeger | aspiers: see how bareon-specs does it. | 09:35 |
| *** tojuvone has quit IRC | 09:35 | |
| AJaeger | or compass-specs | 09:35 |
| *** tojuvone has joined #openstack-infra | 09:35 | |
| AJaeger | Even if you not publish - like compass-specs - I would still build the documents and gate on that. | 09:35 |
| AJaeger | But your call, I just want you to understand the options fully | 09:36 |
| aspiers | did you look at patch set 1? | 09:36 |
| *** daemontool_ has joined #openstack-infra | 09:36 | |
| * AJaeger does now | 09:36 | |
| aspiers | build-only-specs-jobs requires a -docs repo | 09:36 |
| *** daemontool_ has quit IRC | 09:36 | |
| AJaeger | aspiers: no, it does not. | 09:36 |
| AJaeger | It requires a jenkins job. | 09:37 |
| AJaeger | aspiers: read http://jaegerandi.blogspot.de/2016/02/creating-new-test-jobs-in-openstack-ci.html | 09:37 |
| AJaeger | so, define such a job. As I've said: Check how compass-specs does it. | 09:38 |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n847 | 09:38 |
| *** roxanaghe has quit IRC | 09:38 | |
| AJaeger | aspiers: your change one said: Run this job if a new change comes in - but you *nowhere* declared the job and that'S why it failed | 09:38 |
| AJaeger | aspiers: does that help? | 09:39 |
| aspiers | probably | 09:39 |
| aspiers | I'll try after standup | 09:39 |
| aspiers | thanks! | 09:39 |
| aspiers | I wonder why none of this is documented though | 09:39 |
| AJaeger | sure - and if it fails, give me time to review and guide you... | 09:39 |
| aspiers | at least I couldn't find it | 09:39 |
| AJaeger | The infra manual documents a lot - but if you learn new stuff, feel free to enhance it... | 09:40 |
| aspiers | heh | 09:40 |
| aspiers | that happens to me every time :) | 09:40 |
| *** yolanda has quit IRC | 09:42 | |
| *** yolanda has joined #openstack-infra | 09:42 | |
| *** esikachev has joined #openstack-infra | 09:43 | |
| *** akrivoka has joined #openstack-infra | 09:45 | |
| AJaeger | let me do one quick edit for the manual... | 09:46 |
| *** Daisy has quit IRC | 09:46 | |
| *** Daisy has joined #openstack-infra | 09:46 | |
| *** hashar has joined #openstack-infra | 09:49 | |
| *** Daisy has quit IRC | 09:51 | |
| yolanda | back | 09:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Enhance zuul setup section https://review.openstack.org/296348 | 09:52 |
| AJaeger | aspiers: please review ^ | 09:52 |
| AJaeger | yolanda: everything working better now? | 09:52 |
| yolanda | yes, i changed internet provider | 09:53 |
| *** cdent has joined #openstack-infra | 09:53 | |
| AJaeger | Can you now review and approve even faster? ;) | 09:54 |
| yolanda | much faster :) | 09:54 |
| AJaeger | Wow! | 09:55 |
| openstackgerrit | Koteswara Rao Kelam proposed openstack-infra/project-config: Adding networking-hpe project. https://review.openstack.org/295775 | 09:55 |
| AJaeger | yolanda: do you think my patch for infra-manual would have helped aspiers (see backscroll) and others? (296348) | 09:56 |
| *** sc68cal has joined #openstack-infra | 09:56 | |
| *** sshnaidm has quit IRC | 09:56 | |
| yolanda | i lost backlog, need to check eavesdrop | 09:57 |
| *** oanson has quit IRC | 09:58 | |
| *** sdague has joined #openstack-infra | 09:58 | |
| AJaeger | Ah - just last half hour | 09:58 |
| *** dims has joined #openstack-infra | 10:00 | |
| *** sambetts|afk is now known as sambetts | 10:00 | |
| *** gildub has joined #openstack-infra | 10:03 | |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config: Add a microversion-parse project https://review.openstack.org/296046 | 10:04 |
| *** mikelk has quit IRC | 10:05 | |
| *** links has quit IRC | 10:05 | |
| *** mikelk has joined #openstack-infra | 10:05 | |
| *** amitgandhinz has joined #openstack-infra | 10:05 | |
| *** thorst has joined #openstack-infra | 10:08 | |
| *** esikachev has quit IRC | 10:09 | |
| *** zhurong has joined #openstack-infra | 10:11 | |
| *** esikachev has joined #openstack-infra | 10:14 | |
| *** Qiming_ has quit IRC | 10:15 | |
| *** claudiub has joined #openstack-infra | 10:15 | |
| *** thorst has quit IRC | 10:16 | |
| *** claudiub|2 has quit IRC | 10:18 | |
| *** amitgandhinz has quit IRC | 10:19 | |
| *** vgridnev has quit IRC | 10:19 | |
| *** vgridnev has joined #openstack-infra | 10:20 | |
| *** links has joined #openstack-infra | 10:21 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Added validation to tripleo.sh --overcloud-deploy option. https://review.openstack.org/296362 | 10:22 |
| *** bookwar has quit IRC | 10:24 | |
| *** bookwar has joined #openstack-infra | 10:24 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use the mirror server for the centos image and git repos https://review.openstack.org/285257 | 10:25 |
| *** tpsilva has joined #openstack-infra | 10:26 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add execute permissions to promote.sh https://review.openstack.org/296363 | 10:28 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Don't add duplicate entries to metadata.txt https://review.openstack.org/296364 | 10:28 |
| *** nadya has joined #openstack-infra | 10:29 | |
| *** cdent has quit IRC | 10:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:31 | |
| *** nmagnezi has quit IRC | 10:31 | |
| mattt | AJaeger yolanda : hey you two, did either of you see my question earlier? any thoughts on where i can start looking? | 10:32 |
| openstackgerrit | Alexis Lee proposed openstack/requirements: oslo.config>=3.9.0 https://review.openstack.org/291175 | 10:34 |
| AJaeger | mattt: I saw it but have no idea | 10:34 |
| *** jaosorior has quit IRC | 10:34 | |
| yolanda | matt, i lost backlog, can you ask again? | 10:35 |
| *** e0ne has quit IRC | 10:35 | |
| *** jaosorior has joined #openstack-infra | 10:36 | |
| mattt | yolanda: sure, tl;dr is that i'm trying to get a gate job to run locally as it does in jenkins | 10:36 |
| mattt | yolanda: i've spawned the job locally using run-tox.sh, however internal ansible tasks are using pip in tox's ven and in my local testing it uses /usr/local/bin/pip, i have absolutely no idea how that happens and was wondering if you any of you knew so i can get a 100% consistent behaviour | 10:37 |
| yolanda | can you reproduce that venv locally? | 10:38 |
| mattt | yolanda: yeah, tox handles the creation so that is created the same way in gate and CI | 10:39 |
| mattt | sorry, in gate and locally | 10:39 |
| mattt | run-tox.sh sets PYTHON to the venv's python bin, which is fine ... i see that ... but not sure how it's picking up pip in the venv's bin dir | 10:40 |
| *** electrofelix has joined #openstack-infra | 10:41 | |
| *** claudiub|2 has joined #openstack-infra | 10:41 | |
| yolanda | do you have output of the job ran in jenkins? devstacklog, console, etc... can give you more details about it | 10:43 |
| mattt | yolanda: http://logs.openstack.org/65/295265/11/check/gate-openstack-ansible-os_cinder-ansible-func-ubuntu-trusty/d1170a8/console.html#_2016-03-22_13_42_15_517 | 10:44 |
| *** claudiub has quit IRC | 10:44 | |
| mattt | yolanda: specifically: "cmd": "/home/jenkins/workspace/gate-openstack-ansible-os_cinder-ansible-func-ubuntu-trusty/.tox/functional/bin/pip install lxc-python2" | 10:44 |
| *** e0ne has joined #openstack-infra | 10:45 | |
| *** oanson has joined #openstack-infra | 10:45 | |
| AJaeger | mattt: do you want to ssh into such a node and compare that system and yours? | 10:45 |
| mattt | AJaeger: i was just thinking of that, i know in the past it was relatively easy as our jobs ran as root, but this one runs as jenkins i think | 10:46 |
| AJaeger | mattt, yolanda might be able to hold a node for you and add your ssh key - if all else fails ;) | 10:46 |
| mattt | so need to just figure out how that will work | 10:46 |
| mattt | AJaeger: i can jam it in via the review itself, we did that in the past :P | 10:46 |
| AJaeger | mattt: that's a manual setup that only one of the roots here can do - like yolanda | 10:46 |
| AJaeger | mattt: that's evil ;( | 10:47 |
| yolanda | i can hold a node temporarily if there is no better way to debug it | 10:47 |
| mattt | AJaeger: it is :( we had a nasty hp-b4 failure that was killing us | 10:47 |
| yolanda | AJaeger, i wonder if that's related with bindep? | 10:47 |
| mattt | the only way we could resolve was to get into the instance, turned out to be an overlapping IP network that we were trying to create and which only existed in hp-b4 | 10:48 |
| mattt | yolanda: i'll ping you if i need help, thanks in advance for the assistance | 10:48 |
| AJaeger | yolanda: no idea | 10:48 |
| *** yolanda_ has joined #openstack-infra | 10:48 | |
| mattt | AJaeger: just to be clear, i'm not breaking any sort of agreement etc. by ssh into the instance via a review am i ? | 10:49 |
| *** tosky has joined #openstack-infra | 10:49 | |
| mattt | want to make sure i'm not breaking protocol here | 10:49 |
| *** Jeffrey4l has joined #openstack-infra | 10:49 | |
| *** jpr has joined #openstack-infra | 10:49 | |
| *** _degorenko|afk is now known as degorenko | 10:51 | |
| *** sshnaidm has joined #openstack-infra | 10:51 | |
| yolanda_ | matt, we do it only on certain cases where there is no better way to debug, and remove the node as soon as possible | 10:51 |
| *** yolanda has quit IRC | 10:52 | |
| *** yolanda_ is now known as yolanda | 10:52 | |
| *** cdent has joined #openstack-infra | 10:54 | |
| *** tojuvone has quit IRC | 10:54 | |
| *** gildub has quit IRC | 10:54 | |
| *** tojuvone has joined #openstack-infra | 10:54 | |
| *** tojuvone has quit IRC | 10:55 | |
| *** tojuvone has joined #openstack-infra | 10:55 | |
| AJaeger | mattt: we have one time nodes to not have people break in, change stuff, and change other jobs... | 10:55 |
| AJaeger | mattt: I don't know whether there's any agreement for such cases, better ask the rest of the team. | 10:56 |
| mattt | AJaeger: k | 10:57 |
| *** zeih_ has quit IRC | 10:58 | |
| mattt | yolanda: so should i retry the review and then you will put a pause on the instance for me? | 10:58 |
| yolanda | yes, i can do it | 10:58 |
| mattt | yolanda: https://gist.github.com/mattt416/57fe11d36077d2c84fc2 is my pub key | 10:59 |
| *** zhurong has quit IRC | 10:59 | |
| yolanda | matt cool, as soon as you give me the node id, i can hold it | 10:59 |
| mattt | yolanda: excellent, waiting for it to kick off | 11:00 |
| *** gmann has quit IRC | 11:01 | |
| mattt | yolanda: ubuntu-trusty-rax-ord-9003025 | 11:01 |
| *** zhurong has joined #openstack-infra | 11:02 | |
| yolanda | holded, going to add the key | 11:02 |
| mattt | thanks! | 11:02 |
| *** MarkAtwood has joined #openstack-infra | 11:02 | |
| yolanda | try login to 104.239.192.149, using jenkins user | 11:03 |
| *** gmann has joined #openstack-infra | 11:03 | |
| mattt | yolanda: i'm in! i'll let you know as soon as i'm done | 11:04 |
| yolanda | cool, thanks | 11:04 |
| *** markvoelker has joined #openstack-infra | 11:04 | |
| *** dims_ has joined #openstack-infra | 11:05 | |
| *** dims has quit IRC | 11:06 | |
| *** MarkAtwood has quit IRC | 11:07 | |
| *** Qiming has joined #openstack-infra | 11:07 | |
| *** markvoelker has quit IRC | 11:08 | |
| *** Jeffrey4l has quit IRC | 11:10 | |
| *** nmagnezi has joined #openstack-infra | 11:13 | |
| *** amitgandhinz has joined #openstack-infra | 11:15 | |
| vsaienko | neutron infra foks please help to review https://review.openstack.org/#/c/294128/ | 11:15 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add more matches https://review.openstack.org/296377 | 11:16 |
| *** rhallisey has quit IRC | 11:17 | |
| *** Jeffrey4l has joined #openstack-infra | 11:17 | |
| *** rhallisey has joined #openstack-infra | 11:17 | |
| *** achanda has joined #openstack-infra | 11:19 | |
| *** smarcet has joined #openstack-infra | 11:21 | |
| *** roxanaghe has joined #openstack-infra | 11:22 | |
| *** doug-fish has joined #openstack-infra | 11:22 | |
| *** achanda has quit IRC | 11:24 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Enable Murano service as part of congress job https://review.openstack.org/296380 | 11:24 |
| *** roxanaghe has quit IRC | 11:26 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add more matches https://review.openstack.org/296377 | 11:26 |
| *** weshay has quit IRC | 11:27 | |
| *** EricGonczer_ has joined #openstack-infra | 11:28 | |
| *** Jeffrey4l has quit IRC | 11:28 | |
| *** amitgandhinz has quit IRC | 11:29 | |
| *** ociuhandu has quit IRC | 11:29 | |
| *** ldnunes has joined #openstack-infra | 11:29 | |
| *** Nikolay_St has joined #openstack-infra | 11:31 | |
| *** esikachev has quit IRC | 11:34 | |
| *** thorst has joined #openstack-infra | 11:35 | |
| *** jordanP has joined #openstack-infra | 11:36 | |
| *** tojuvone has quit IRC | 11:39 | |
| *** tojuvone has joined #openstack-infra | 11:40 | |
| AJaeger | sdague: could you review as devstack expert a new job that gets added to devstack and tempest, please? https://review.openstack.org/292894 | 11:43 |
| *** Jeffrey4l has joined #openstack-infra | 11:47 | |
| sdague | AJaeger: reviewed | 11:50 |
| evgenyl | AJaeger pabelanger fungi: So I've fixed my tests using bindep https://review.openstack.org/#/c/296337/ , works fine except one problem, not sure how it should be solved properly, but if package doesn't use DB (but uses standard gates to run tests), it fails with "mysqladmin: not found" and "unknown user: postgres" errors, so as a workaround, I added mysql | 11:50 |
| evgenyl | and postgres as dependencies for my project. | 11:50 |
| AJaeger | sdague: thanks. See also https://review.openstack.org/#/q/owner:%22Jamie+Lennox%22+status:open+project:openstack-infra/project-config | 11:51 |
| AJaeger | evgenyl: yes, known issue. That's why it's still experimental ;) | 11:51 |
| evgenyl | AJaeger: cool, thanks for your help :) | 11:52 |
| AJaeger | evgenyl: REad http://lists.openstack.org/pipermail/openstack-dev/2016-March/090052.html on what we need to do first | 11:52 |
| AJaeger | evgenyl: once https://review.openstack.org/#/c/296039/ is in, you can remove the db packages | 11:52 |
| evgenyl | AJaeger: ok | 11:53 |
| AJaeger | evgenyl: change looks fine. | 11:53 |
| sdague | AJaeger: honestly, I don't think building up context from a selection of reviews is the right approach here. There should be some document that's the coherent plan of how we get from x -> y. I feel like I end up reverting about 1/3 of all the keystone v3 work because it wasn't thought through. And am just really not excited to end up there again. | 11:56 |
| *** zeih has joined #openstack-infra | 11:57 | |
| *** Jeffrey4l has quit IRC | 11:58 | |
| *** david-lyle_ has joined #openstack-infra | 11:58 | |
| *** ociuhandu has joined #openstack-infra | 11:59 | |
| *** esikachev has joined #openstack-infra | 12:00 | |
| *** david-lyle has quit IRC | 12:00 | |
| AJaeger | sdague: understood. | 12:01 |
| *** Daisy has joined #openstack-infra | 12:01 | |
| *** salv-orl_ has joined #openstack-infra | 12:02 | |
| *** weshay has joined #openstack-infra | 12:02 | |
| *** salv-orlando has quit IRC | 12:04 | |
| *** |-paul-| has joined #openstack-infra | 12:07 | |
| *** dprince has joined #openstack-infra | 12:07 | |
| *** _amrith_ is now known as amrith | 12:09 | |
| *** annegentle has joined #openstack-infra | 12:10 | |
| *** jcoufal has joined #openstack-infra | 12:11 | |
| *** pblaho has quit IRC | 12:11 | |
| *** lucasagomes is now known as lucas-hungry | 12:12 | |
| *** aysyd has joined #openstack-infra | 12:13 | |
| *** jaypipes has joined #openstack-infra | 12:15 | |
| *** asettle has quit IRC | 12:19 | |
| wznoinsk | hi | 12:20 |
| *** pblaho has joined #openstack-infra | 12:20 | |
| *** asettle has joined #openstack-infra | 12:20 | |
| *** markvoelker has joined #openstack-infra | 12:20 | |
| *** yamamoto_ has quit IRC | 12:20 | |
| wznoinsk | would anyone know is there a way to invoke a recheck and have the instance held by nodepool for some time? I couldn't find any code supporting that in nodepool and none of the pipelines that might be configured this way neither anything in zuul's source code... | 12:22 |
| *** annegentle has quit IRC | 12:23 | |
| *** asettle has quit IRC | 12:24 | |
| *** amitgandhinz has joined #openstack-infra | 12:25 | |
| *** Jeffrey4l has joined #openstack-infra | 12:25 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Allow an update of the SearchCriteriaController to be triggered https://review.openstack.org/295451 | 12:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Improve the worklist UI https://review.openstack.org/295450 | 12:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a resource abstraction for worklist filters https://review.openstack.org/295453 | 12:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Improve the tag-complete directive https://review.openstack.org/295452 | 12:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Allow filters to be set on automatic worklists https://review.openstack.org/295454 | 12:26 |
| *** yaume has joined #openstack-infra | 12:27 | |
| *** rfolco has joined #openstack-infra | 12:28 | |
| *** edmondsw has joined #openstack-infra | 12:28 | |
| *** smarcet has quit IRC | 12:28 | |
| hashar | wznoinsk: there is a command line argument to nodepool to hold an instance and prevent it from being deleted | 12:29 |
| hashar | wznoinsk: nodepool hold <id> | 12:29 |
| *** Daisy has quit IRC | 12:29 | |
| hashar | that will mark it with status 'hold' and shows up as such in 'nodepool list' | 12:29 |
| hashar | but you need access on nodepool :-} | 12:30 |
| *** Daisy has joined #openstack-infra | 12:30 | |
| wznoinsk | hashar, yes, I was wondering is there a way to 'invoke' that behaviour by specially formated comment (didn't find anything in os infra's project-config tho) | 12:31 |
| vincentll | our project got accepted on OpenStack CI (thanks!), could someone on the infra team add me to the groups: synergy-core & synergy-release? my handle is vincentll | 12:32 |
| *** krtaylor has quit IRC | 12:32 | |
| *** pradk is now known as pradk_ | 12:33 | |
| *** ociuhandu has quit IRC | 12:34 | |
| *** Daisy has quit IRC | 12:34 | |
| *** ociuhandu has joined #openstack-infra | 12:35 | |
| *** dizquierdo has quit IRC | 12:35 | |
| *** piet has joined #openstack-infra | 12:36 | |
| hashar | zaro: regarding git clone performance, github has an article about generating bitmaps to speed things up http://githubengineering.com/counting-objects/ . Apparently they proposed a patch to JGIt to support it | 12:36 |
| *** oanson has quit IRC | 12:36 | |
| *** annegentle has joined #openstack-infra | 12:37 | |
| mattt | yolanda: didn't figure it out :( but have to go into a meeting for an hour, so probably best to nuke that instance | 12:38 |
| *** amitgandhinz has quit IRC | 12:38 | |
| *** Jeffrey4l has quit IRC | 12:38 | |
| *** nelsnelson has joined #openstack-infra | 12:39 | |
| *** Jeffrey4l has joined #openstack-infra | 12:40 | |
| *** tojuvone has quit IRC | 12:41 | |
| *** tojuvone has joined #openstack-infra | 12:42 | |
| *** tojuvone has quit IRC | 12:42 | |
| *** tojuvone has joined #openstack-infra | 12:42 | |
| hashar | wznoinsk: not that I know of | 12:44 |
| dims_ | good point AJaeger on that oslo.messaging one. made the job voting now | 12:44 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Use standard python3-jobs definition for oslo.messaging https://review.openstack.org/296374 | 12:44 |
| hashar | wznoinsk: and I am pretty sure infra dislikes the idea of making Zuul to recognize more commands from gerrit comments beside 'recheck' | 12:44 |
| *** yaume_ has joined #openstack-infra | 12:45 | |
| *** |-paul-| has quit IRC | 12:45 | |
| *** krtaylor has joined #openstack-infra | 12:46 | |
| *** Jeffrey4l has quit IRC | 12:47 | |
| *** yaume has quit IRC | 12:47 | |
| *** baoli has joined #openstack-infra | 12:48 | |
| yolanda | ok i'll remove the instance | 12:49 |
| *** smarcet has joined #openstack-infra | 12:50 | |
| smarcet | fungi: morning | 12:50 |
| *** MarkAtwood has joined #openstack-infra | 12:51 | |
| *** ddieterly has joined #openstack-infra | 12:52 | |
| *** ociuhandu has quit IRC | 12:52 | |
| *** ociuhandu has joined #openstack-infra | 12:53 | |
| *** ddieterly is now known as ddieterly[away] | 12:53 | |
| *** mtanino has quit IRC | 12:53 | |
| wznoinsk | hashar, thanks for the info, I'm thinking of such an option and heard from someone it may be available upstream already so wanted to check, it maybe handy for our 3rdparty ci so may try to implement it myself then | 12:54 |
| *** MarkAtwood has quit IRC | 12:55 | |
| *** kgiusti has joined #openstack-infra | 12:55 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to scan project repo for new bug fixes in master https://review.openstack.org/296420 | 12:55 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs of specified importance https://review.openstack.org/296421 | 12:56 |
| *** EricGonczer_ has quit IRC | 12:57 | |
| *** zeih has quit IRC | 12:58 | |
| *** markus_z is now known as markus_z_meeting | 12:59 | |
| openstackgerrit | Zhijiang Hu proposed openstack-infra/project-config: [new project] Create a OpenStack related project "daisycloud-core" https://review.openstack.org/296117 | 13:00 |
| *** rlandy has joined #openstack-infra | 13:01 | |
| *** links has quit IRC | 13:01 | |
| *** sfinucan has quit IRC | 13:02 | |
| mattt | yolanda: thanks for your help there | 13:02 |
| *** rossella_s has quit IRC | 13:03 | |
| *** ddieterly[away] has quit IRC | 13:03 | |
| *** rossella_s has joined #openstack-infra | 13:03 | |
| *** sfinucan has joined #openstack-infra | 13:05 | |
| *** david-lyle_ is now known as david-lyle | 13:06 | |
| *** jsavak has joined #openstack-infra | 13:07 | |
| *** roxanaghe has joined #openstack-infra | 13:10 | |
| *** amrith is now known as _amrith_ | 13:10 | |
| *** yamamoto has joined #openstack-infra | 13:12 | |
| *** lucas-hungry is now known as lucasagomes | 13:14 | |
| *** bdemers has joined #openstack-infra | 13:14 | |
| *** roxanaghe has quit IRC | 13:14 | |
| *** amitgandhinz has joined #openstack-infra | 13:14 | |
| *** amitgandhinz has quit IRC | 13:15 | |
| *** amitgandhinz has joined #openstack-infra | 13:15 | |
| *** baoli_ has joined #openstack-infra | 13:16 | |
| *** sfinucan has quit IRC | 13:17 | |
| *** mriedem has joined #openstack-infra | 13:18 | |
| *** jpr has quit IRC | 13:18 | |
| *** zeih has joined #openstack-infra | 13:19 | |
| *** baoli has quit IRC | 13:19 | |
| *** achanda has joined #openstack-infra | 13:22 | |
| *** vikram has quit IRC | 13:22 | |
| akrivoka | hi, could somebody from infra core please add the 'tripleo-core' group to the 'tripleo-ui-core'? https://review.openstack.org/#/admin/groups/1320,members | 13:22 |
| *** oanson has joined #openstack-infra | 13:22 | |
| *** sfinucan has joined #openstack-infra | 13:23 | |
| *** tonytan4ever has joined #openstack-infra | 13:24 | |
| *** moravec has quit IRC | 13:24 | |
| yolanda | akrivoka, i can add one member to the group, the person that created the project, and then this person can add others, is that fine for you? can you link me the change of project creation? | 13:26 |
| *** MCoLo__ has quit IRC | 13:26 | |
| akrivoka | yolanda: sure, one sec | 13:26 |
| *** achanda has quit IRC | 13:27 | |
| akrivoka | yolanda: https://review.openstack.org/#/c/291227/ | 13:27 |
| *** moravec has joined #openstack-infra | 13:27 | |
| akrivoka | yolanda: I created the project, though I will not be in the core reviewer group for the project | 13:27 |
| yolanda | akriovoka, i can add you , then you add others and remove | 13:28 |
| sbelous | sdague: Hi Sean, can you please again take a look at my patch to devstack when you will have a time? https://review.openstack.org/#/c/286084/ | 13:28 |
| akrivoka | yolanda: that's fine by me | 13:28 |
| akrivoka | yolanda: thanks! | 13:28 |
| *** darvon has quit IRC | 13:29 | |
| yolanda | akrivoka, done | 13:29 |
| akrivoka | yolanda: awesome, thanks so much! | 13:29 |
| yolanda | glad to help | 13:29 |
| akrivoka | :) | 13:29 |
| *** darvon has joined #openstack-infra | 13:31 | |
| *** regXboi has joined #openstack-infra | 13:32 | |
| *** rbrndt has joined #openstack-infra | 13:33 | |
| *** daemontool_ has joined #openstack-infra | 13:33 | |
| *** daemontool has quit IRC | 13:33 | |
| *** lezbar has quit IRC | 13:33 | |
| *** lezbar has joined #openstack-infra | 13:35 | |
| *** jamesmcarthur has joined #openstack-infra | 13:36 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs of specified importance https://review.openstack.org/296421 | 13:36 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to scan project repo for new bug fixes in master https://review.openstack.org/296420 | 13:36 |
| *** vgridnev has quit IRC | 13:37 | |
| *** knikolla has joined #openstack-infra | 13:39 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Add mitaka stable branch jobs https://review.openstack.org/295499 | 13:42 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 13:42 |
| *** mtanino has joined #openstack-infra | 13:44 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs of specified importance https://review.openstack.org/296421 | 13:45 |
| *** piet has quit IRC | 13:45 | |
| *** vgridnev has joined #openstack-infra | 13:48 | |
| *** gordc has joined #openstack-infra | 13:49 | |
| *** hashar has quit IRC | 13:49 | |
| fungi | smarcet: so i think the plan is this: 1. temporarily disable puppet for openstackid.org, 2. merge a change to use $::hostname instead of $::fqdn in the laravel template, 3. correct /etc/hostname on openstackid-dev and make sure it's working there, 4. manually change the laravel config and hostname on the production openstackid.org and make sure it continues to work, 5. reenable puppet for | 13:51 |
| fungi | openstackid.org again | 13:51 |
| pabelanger | morning | 13:51 |
| anteaya | ttx: do you want to merge https://review.openstack.org/#/c/295499/4 now or wait for the definition for propose-requirements-constraints-mitaka? | 13:51 |
| anteaya | morning pabelanger | 13:51 |
| fungi | smarcet: is it just the $vhost_name we need to adjust in the openstackid class? | 13:52 |
| fungi | smarcet: or is it $id_hostname? | 13:52 |
| smarcet | fungi: i think that $id_hostname | 13:52 |
| smarcet | $vhost_name should remain as it | 13:52 |
| fungi | i'm checking to see where those end up in the templates now, but looks like it yes | 13:53 |
| mriedem | AJaeger: do you know if anyone is working on stable bitrot jobs for mitaka? | 13:53 |
| anteaya | mriedem: https://review.openstack.org/#/c/295499/4 | 13:53 |
| mriedem | sweet | 13:53 |
| fungi | smarcet: agreed, $id_hostname is the one that ends up in environment.php, while $vhost_name just gets used in the apache vhost config | 13:54 |
| ttx | anteaya: maybe wait for clarkb to let us know if he'd rather define that job in the same run or as a subsequent change | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Always do network interface introspection https://review.openstack.org/296450 | 13:54 |
| ttx | that may be what he had in mind | 13:54 |
| fungi | smarcet: i'll go ahead and propose the change for this now and also disable puppet for openstackid.org | 13:54 |
| *** mriedem is now known as mriedem_meeting | 13:55 | |
| smarcet | fungi: roger | 13:55 |
| *** sfinucan has quit IRC | 13:55 | |
| *** annegentle has quit IRC | 13:55 | |
| anteaya | ttx: will wait | 13:56 |
| *** piet has joined #openstack-infra | 13:57 | |
| *** esikachev has quit IRC | 13:58 | |
| *** vdrok has joined #openstack-infra | 13:59 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackid: Default to hostname not fqdn for laravel env https://review.openstack.org/296453 | 13:59 |
| *** BigWillie has joined #openstack-infra | 13:59 | |
| fungi | smarcet: ^ | 13:59 |
| smarcet | fungi: roger | 13:59 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs with specific tag https://review.openstack.org/296454 | 13:59 |
| *** sfinucan has joined #openstack-infra | 14:00 | |
| jroll | gerrit seems to be grinding to a halt :( | 14:00 |
| ttx | gerrit slow | 14:00 |
| TheJulia | I was just going to say the same | 14:00 |
| vdrok | same for me :) | 14:00 |
| ttx | damn, jinxed by jroll | 14:00 |
| jroll | :D | 14:00 |
| pabelanger | jeblair: fungi: Starting on OpenAFS administration. Just confirming, I'm okay to create an kerberos principal account. Also, does it need to be an admin principal? | 14:00 |
| ttx | who needs nagios when you have 2500 devs | 14:01 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 14:01 |
| *** pradk has joined #openstack-infra | 14:01 | |
| anteaya | traffic spike of some kind | 14:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:01 | |
| *** hashar has joined #openstack-infra | 14:02 | |
| *** _amrith_ is now known as amrith | 14:02 | |
| anteaya | eth0 outbound traffic increase | 14:03 |
| anteaya | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=45&rra_id=all | 14:03 |
| anteaya | infra-root can someone take a peek at gerrit and share what they are seeing? | 14:04 |
| *** dims has joined #openstack-infra | 14:04 | |
| *** dims_ has quit IRC | 14:04 | |
| anteaya | a line up of unhappy devs are waiting to hear from you | 14:04 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Temporarily disable Puppet for openstackid.org https://review.openstack.org/296456 | 14:05 |
| fungi | smarcet: and that ^ | 14:05 |
| *** achanda has joined #openstack-infra | 14:05 | |
| fungi | anteaya: i'll take a look in the logs | 14:05 |
| anteaya | fungi: thank you | 14:05 |
| anteaya | fungi: of course the normal cpu gc issue is suspect, but the odd thing I saw was the traffic spike | 14:06 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackid: Default to hostname not fqdn for laravel env https://review.openstack.org/296453 | 14:06 |
| *** achanda has quit IRC | 14:07 | |
| *** salv-orl_ has quit IRC | 14:07 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Improve the worklist UI https://review.openstack.org/295450 | 14:08 |
| *** openstack has joined #openstack-infra | 14:22 | |
| fungi | something to keep an eye out for on future replacements of other servers | 14:22 |
| *** nmagnezi has quit IRC | 14:23 | |
| *** aspiers has joined #openstack-infra | 14:23 | |
| mordred | fungi: nice! | 14:23 |
| *** nmagnezi has joined #openstack-infra | 14:23 | |
| fungi | mordred: also not sure if you saw my updates after you dropped off last night. your changes mostly worked, but we still get some errors at the end and i think there are also cache update issues | 14:23 |
| jeblair | i'm spinning up a server now, anything i should look for? | 14:24 |
| fungi | mordred: oh, though the errors may be because it was not running the groups expansion fixes in my local clone | 14:24 |
| fungi | jeblair: we want to make sure mordred's fixes from last night are merged | 14:24 |
| fungi | digging them up now | 14:25 |
| jeblair | fungi: whoops :) | 14:25 |
| mordred | fungi: oh hehe. :) | 14:25 |
| *** ddieterly is now known as ddieterly[away] | 14:25 | |
| *** jed561 has quit IRC | 14:25 | |
| mordred | fungi: there is a larger thing here I think we do want to talk about design a little deeper - perhaps in austin | 14:25 |
| jeblair | mordred: 1 sentence summary of that thing? | 14:26 |
| fungi | jeblair: 296128 and 296450 | 14:26 |
| fungi | oh, hrm, 296128 hit a timeout after it got approved... rechecking it now | 14:27 |
| *** annegentle has joined #openstack-infra | 14:27 | |
| *** sdake has joined #openstack-infra | 14:27 | |
| jeblair | fungi: beat you | 14:27 |
| fungi | cancelling mine ;) | 14:27 |
| pabelanger | jeblair: just to confirm, I'll use 2010 as the UID for http://docs.openstack.org/infra/system-config/afs.html#adding-a-user which came from puppet manifests | 14:27 |
| mordred | jeblair: the interaction between launch-node, ansible and puppet and how group membership works | 14:27 |
| mordred | jeblair: because we currently have a situaiton where nodes will not get group variables at launch-node time | 14:28 |
| fungi | oh, 296450 was something else | 14:28 |
| mattt | hi all, can someone save ubuntu-trusty-ovh-gra1-9012324 for me and throw my key (https://gist.github.com/mattt416/57fe11d36077d2c84fc2) on it please ? | 14:28 |
| *** akrivoka has quit IRC | 14:28 | |
| fungi | i guess mordred's other change already merged | 14:28 |
| mattt | yolanda helped me earlier but not sure if they're about any more | 14:28 |
| jeblair | pabelanger: correct | 14:28 |
| openstackgerrit | Merged openstack-infra/project-config: blazar,cerberus,kite: Use db setup for python jobs https://review.openstack.org/294949 | 14:29 |
| openstackgerrit | Merged openstack-infra/project-config: subunit2sql: Use db setup for python jobs https://review.openstack.org/295044 | 14:29 |
| *** IlyaG has joined #openstack-infra | 14:30 | |
| *** dprince has quit IRC | 14:30 | |
| fungi | jeblair: aha, the other change i was trying to think of was 296127 (not 296450) but it did already merge earlier | 14:31 |
| *** piet has quit IRC | 14:31 | |
| jeblair | mordred: i added your topic to the summit planning etherpad :) | 14:32 |
| fungi | jeblair: also i need to propose an update to the readme since the reverse dns commands are now incomplete (missing required environment variables). coming up in moments | 14:32 |
| pabelanger | jeblair: the afs docs are not clear, but I _think_ I should be running the commands as root, as well as kinit principal admin? | 14:32 |
| *** kushal has quit IRC | 14:32 | |
| pabelanger | I tried as local user seen permission denied from pts | 14:33 |
| jeblair | pabelanger: which commands? | 14:33 |
| pabelanger | jeblair: pts createuser $USERNAME -id UID | 14:33 |
| *** tongli has joined #openstack-infra | 14:34 | |
| pabelanger | I am currently at step: http://docs.openstack.org/infra/system-config/afs.html#adding-a-user | 14:35 |
| jeblair | pabelanger: that should be able to run on any afs client as a regular user kinit principal admin; or on an afs server as root with '-localauth' appended without kinit | 14:35 |
| jroll | krotscheck: are there openstack projects currently using node.js, or is everything frontend so far? | 14:35 |
| anteaya | fungi: did you get a chance to look at why the volume of logs on jenkins 03 and 05 was so much more than the other jenkinsii last night? | 14:36 |
| *** sc68cal has quit IRC | 14:36 | |
| anteaya | mattt: can you give us some background on why you want a node held? | 14:36 |
| jeblair | fungi, mordred: launch-node appeared to dtrt with hostnames | 14:37 |
| fungi | jeblair: excellent | 14:37 |
| *** piet has joined #openstack-infra | 14:38 | |
| mordred | jeblair: woot! | 14:38 |
| *** MarkAtwood has joined #openstack-infra | 14:39 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Use environment file in sample reverse DNS command https://review.openstack.org/296494 | 14:39 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-gerrit: Order of the classes parameters is refactored https://review.openstack.org/294728 | 14:39 |
| fungi | mordred: jeblair: not the readme actually ^ | 14:39 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 14:40 |
| *** IlyaG has quit IRC | 14:40 | |
| *** andymaier has joined #openstack-infra | 14:40 | |
| *** EricGonczer_ has quit IRC | 14:41 | |
| jeblair | fungi: i just now used openstackci-rax-dfw, but diff says they are the same, so +2 | 14:41 |
| mattt | anteaya: there is some behaviour we're seeing in our gate job which i cannot replicate locally, see https://gist.github.com/mattt416/165aa6393c3b9a92ba9d ... basically in the gate the venv's PATH seems to be retained when ansible switches to the root user to execute the task, however when i run locally it resets PATH so we get different behaviour | 14:41 |
| pabelanger | jeblair: from an afs client, as regular user kinit principal admin I get the following error: http://paste.openstack.org/show/491594 | 14:41 |
| anteaya | mattt: thank you that helps | 14:42 |
| jeblair | pabelanger: output of 'klist'? | 14:42 |
| mattt | anteaya: that instance has since died off, can i retry this job and grab a fresh instance? | 14:42 |
| *** sfinucan has quit IRC | 14:42 | |
| anteaya | mattt: make contact with an infra-root first | 14:42 |
| *** zhurong has quit IRC | 14:42 | |
| pabelanger | jeblair: http://paste.openstack.org/show/491596/ | 14:42 |
| anteaya | so you have an agreement from someone that they will help you | 14:42 |
| jeblair | pabelanger: run 'aklog' | 14:42 |
| *** tonytan4ever has quit IRC | 14:42 | |
| anteaya | mattt: it is every run or intermittent? | 14:43 |
| *** mriedem_meeting is now known as mriedem | 14:43 | |
| pabelanger | jeblair: nothing outputted | 14:43 |
| mattt | anteaya: every run | 14:43 |
| jeblair | pabelanger: now run 'klist' again | 14:43 |
| *** sfinucan has joined #openstack-infra | 14:43 | |
| *** tonytan4ever has joined #openstack-infra | 14:43 | |
| pabelanger | http://paste.openstack.org/show/491597/ | 14:43 |
| mattt | anteaya: we have a larger project (openstack-ansible) that runs everything as root, there there are no issues, however on these roles we run ansible as the jenkins user, and that's where the issue lies i believe | 14:43 |
| anteaya | mattt: okay hopefully an infra-root will take a minute to grab your key and tell you when to trigger the job so they can add you | 14:43 |
| *** Swami has joined #openstack-infra | 14:43 | |
| mattt | anteaya: no rush, appreciate the help | 14:43 |
| *** MarkAtwood has quit IRC | 14:44 | |
| anteaya | mattt: sure, that is my recommendation, follow along with the conversations and see when one is finishing up, have the url to your key handy | 14:44 |
| jeblair | pabelanger: try the pts command again now; the second entry in klist is your afs service ticket. the first one is the kerberos ticket granting ticket; aklog uses it to request a service ticket for afs. the service ticket is what actually gets handed over to the afs servers to authn you. it's just like keystone ;) | 14:45 |
| openstackgerrit | Merged openstack-infra/openstackid: * fix on access token revoke * improved tokens generation * fixed some typos https://review.openstack.org/296461 | 14:45 |
| mattt | anteaya: i've tried running my job locally using an unprivileged user and even spawned it with run-tox.sh, but cannot replicate that behaviour | 14:45 |
| mattt | but will wait for an infra-root, thanks ! | 14:45 |
| *** dizquierdo has joined #openstack-infra | 14:45 | |
| mordred | Shrews: ping when you're awake | 14:45 |
| anteaya | mattt: yup, that was what we needed to see, an effort to replicate locally, thank you | 14:46 |
| anteaya | mordred: are you able to help mattt, he would like access to a node when it runs a certain job | 14:46 |
| Shrews | mordred: here, just coffee'ing with TheJulia | 14:46 |
| pabelanger | jeblair: right. So the libproto message are now gone from pts createuser, but I still have: pts: Permission denied ; unable to create user pabelanger with id 2010 | 14:46 |
| mordred | anteaya: not this instant | 14:46 |
| anteaya | mordred: okay sorry to bother | 14:46 |
| *** david-lyle_ is now known as david-lyle | 14:47 | |
| jeblair | pabelanger: ah, you're bootstrapping yourself. i believe you will need to run it on an afs server as root with -localauth and without using kinit. | 14:47 |
| jeblair | pabelanger: sorry i didn't think of that earlier. :) | 14:47 |
| pabelanger | jeblair: Ah, okay. I'll update that docs | 14:47 |
| *** sdake_ has joined #openstack-infra | 14:48 | |
| AJaeger | anteaya: nothing special planned, just family time ;) | 14:48 |
| *** akshai has joined #openstack-infra | 14:48 | |
| anteaya | AJaeger: sounds nice :) | 14:48 |
| pabelanger | jeblair: worked, thanks | 14:48 |
| *** sdake has quit IRC | 14:49 | |
| jeblair | pabelanger: if *i* were making your user, i could do that on a client with kinit since my user already exists in afs. sorry i didn't think of that earlier. afs has its own authZ (not authN) database -- that's what the PTS database is, and of course you are creating your own entry it it, so you are not yet authorized to modify it :) | 14:49 |
| dougwig | AJaeger: does family time mean that it'll take you like 4 minutes to get to every infra review, instead of the standard 30 seconds? :) | 14:49 |
| *** ddieterly[away] is now known as ddieterly | 14:49 | |
| jeblair | pabelanger: so the -localauth thing bypasses authz for cases like this | 14:49 |
| AJaeger | dougwig: something like that ;) | 14:49 |
| *** zeih has quit IRC | 14:50 | |
| anteaya | dougwig: ha ha ha | 14:50 |
| *** zzzeek has quit IRC | 14:50 | |
| *** hichihara has joined #openstack-infra | 14:51 | |
| eharney | hi, i've created a devstack-plugin-nfs project via https://review.openstack.org/#/c/289589/ , but i need to be added to -core there | 14:51 |
| fungi | jeblair: pabelanger: on the other hand, bootstrapping your account yourself is a good learning experience. we need more people with intimate familiarity in those tools | 14:52 |
| jeblair | yep | 14:52 |
| pabelanger | jeblair: so, do I need 2 pts users? one for pabelanger and one for pabelanger.admin? | 14:52 |
| jeblair | pabelanger: yes | 14:52 |
| pabelanger | jeblair: ok | 14:52 |
| AJaeger | dougwig: could you review https://review.openstack.org/#/c/282591/ , please? That needs a neutron liason to bless... | 14:52 |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn jobs https://review.openstack.org/258522 | 14:52 |
| *** Jeffrey4l has joined #openstack-infra | 14:52 | |
| *** sdake has joined #openstack-infra | 14:52 | |
| *** sdake_ has quit IRC | 14:53 | |
| pabelanger | jeblair: okay, pt listentries shows both pabelanger and pabelanger.admin | 14:54 |
| pabelanger | jeblair: anyway to list the settings with bos? | 14:54 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for httpretty liberty unicode bug 1560808 https://review.openstack.org/296510 | 14:54 |
| openstack | bug 1560808 in OpenStack-Gate "stable/liberty: gate-requirements-integration-dsvm fails with due to UnicodeDecodeError in httpretty" [Undecided,Confirmed] https://launchpad.net/bugs/1560808 | 14:54 |
| anteaya | eharney: yes both groups have been created, just waiting on an infra-root with a spare moment to add you: https://review.openstack.org/#/admin/groups/?filter=devstack-plugin-nfs- | 14:54 |
| eharney | anteaya: ah great, thanks | 14:55 |
| dougwig | AJaeger: looking | 14:55 |
| *** sc68cal has joined #openstack-infra | 14:55 | |
| jeblair | pabelanger: i think there's a 'bos listuser' command. "bos help" gives a summary command list. | 14:55 |
| pabelanger | jeblair: okay, thanks | 14:55 |
| *** roxanaghe has joined #openstack-infra | 14:56 | |
| pabelanger | looks like I did it correctly | 14:56 |
| jeblair | er, bos listusers; but you can shorten it to any length that still uniquely identifies the command | 14:56 |
| *** jsavak has quit IRC | 14:56 | |
| jeblair | (so 'bos listu' works, for instance, but not 'bos list') | 14:56 |
| jeblair | (this is true for all afs commands) | 14:56 |
| pabelanger | right | 14:57 |
| openstackgerrit | Ryan Brandt proposed openstack-infra/project-config: Use TEMPEST_SERVICES variable to trigger monasca tempest tests https://review.openstack.org/285577 | 14:59 |
| *** yamahata has joined #openstack-infra | 14:59 | |
| *** akaszuba has quit IRC | 14:59 | |
| *** annegentle has quit IRC | 14:59 | |
| jeblair | pabelanger: i have to run to breakfast; before you create volumes, you might poke around with the 'vos listvldb' command, as well as 'vos examine' and the 'fs examine' and 'fs listquota' commands on the filesystem | 15:00 |
| pabelanger | jeblair: perfect, doing that now | 15:00 |
| *** roxanaghe has quit IRC | 15:00 | |
| *** SotK_ is now known as SotK | 15:00 | |
| *** kushal has joined #openstack-infra | 15:00 | |
| smarcet | fungi: are we good to swap ips and start using new server ? | 15:01 |
| *** dizquierdo is now known as dizquierdo_afk | 15:01 | |
| *** bryan_att has joined #openstack-infra | 15:01 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 15:01 |
| yolanda | hi matt, i am around again, can i help? | 15:01 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-gerrit: Order of the classes parameters is refactored https://review.openstack.org/294728 | 15:02 |
| *** zzzeek has joined #openstack-infra | 15:02 | |
| *** gordc has quit IRC | 15:03 | |
| *** jsavak has joined #openstack-infra | 15:03 | |
| fungi | smarcet: i was going to get the current series of changes merged, the hostname workaround confirmed safe on openstackid-dev, manually applying it to the current production server and turn puppet back on again so i can rebuild the replacement and confirm it comes up correctly the first time around, then we can switch over to it | 15:04 |
| smarcet | fungi: roger thx for info ;) | 15:04 |
| fungi | smarcet: we've got a few other puppet changes ahead of those in the gate at the moment but should be merging any minute | 15:04 |
| smarcet | fungi: cool | 15:05 |
| *** mrmartin has joined #openstack-infra | 15:05 | |
| *** tonytan4ever has quit IRC | 15:06 | |
| anteaya | yolanda: if you have a moment can you dig up openstackstatus? it seems to have disappeared | 15:06 |
| *** tonytan4ever has joined #openstack-infra | 15:06 | |
| fungi | smarcet: basically as soon as we see openstackid-dev.openstack.org break, i'll manually correct the hostname on it and make sure it comes up properly, then can do the same on openstackid.org | 15:06 |
| fungi | trying to keep the impact to production as brief as possible | 15:06 |
| smarcet | fungi: understood ;) | 15:07 |
| yolanda | anteaya, no statusbot? | 15:07 |
| anteaya | yolanda: I don't see it | 15:07 |
| fungi | yolanda: apparently openstackstatus timed out in a netslipt/freenode disruption an hour or so ago | 15:07 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again https://review.openstack.org/288915 | 15:07 |
| yolanda | ok i can look at it | 15:08 |
| anteaya | wznoinsk: we don't have a command that the public can access to hold a node | 15:08 |
| fungi | openstack is still here, so statusbot may just need a restart to notice it needs to reconnect | 15:08 |
| anteaya | wznoinsk: we have a certain quota from our host companies and have to ensure that nodes get rotated frequently, so we wouldn't give the public access to such a command | 15:09 |
| anteaya | yolanda: thank you | 15:09 |
| cj | hey folks | 15:09 |
| *** EricGonczer_ has joined #openstack-infra | 15:09 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 15:09 |
| cj | I'm getting no feedback on my gerritbot patch | 15:09 |
| jeblair | pabelanger: i'm back whenever you are | 15:10 |
| cj | who can I pester to have my patch reviewed? | 15:10 |
| yolanda | bot restarted | 15:11 |
| anteaya | yolanda: thank you | 15:11 |
| pabelanger | jeblair: okay, so I think we need shorter volume names: vos: the name of the root volume mirror.wheel.centos7x64 exceeds the size limit of 22 | 15:11 |
| anteaya | cj: we are fairly busy getting services in place in preparation for summit | 15:11 |
| pabelanger | mirror.wheel.trustyx64 is 22 chars on the nose | 15:11 |
| anteaya | cj: the best way to get attention on your patch is to review other patches | 15:11 |
| *** openstackstatus has joined #openstack-infra | 15:12 | |
| *** ChanServ sets mode: +v openstackstatus | 15:12 | |
| yolanda | here it is | 15:12 |
| anteaya | yay | 15:12 |
| anteaya | thank you | 15:12 |
| anteaya | yolanda: have you a moment to add eharney to these newly created groups? https://review.openstack.org/#/c/289589/7/gerrit/acls/openstack/devstack-plugin-nfs.config | 15:13 |
| openstackgerrit | Merged openstack-infra/project-config: Adding branch creation priv to group networking-brocade-release https://review.openstack.org/292635 | 15:13 |
| yolanda | will do | 15:14 |
| fungi | we're still seeing a lot of jobs timing out on remote git operations, i've had several strike my changes, and i just aborted one in the gate which had been sitting at "updating git remote" on a fairly small repo for over 20 minutes | 15:14 |
| anteaya | yolanda: thank you | 15:14 |
| yolanda | no problem | 15:14 |
| openstackgerrit | Merged openstack-infra/project-config: Add networking-generic-switch-dsvm job https://review.openstack.org/294128 | 15:14 |
| anteaya | fungi: was that the issue jeblair was working on yesterday with the operating system versions? | 15:14 |
| openstackgerrit | Merged openstack-infra/project-config: Move networking-odl rally job into check. https://review.openstack.org/282591 | 15:14 |
| yolanda | done | 15:15 |
| jeblair | pabelanger: yes. we'll need to shorten the last component, but leave the others the same. | 15:15 |
| anteaya | yolanda: thanks, eharney you should be good to go | 15:15 |
| fungi | not sure why this has cropped back up so recently but it doesn't seem to be on any particular distro or provider | 15:15 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-openstackci: Order of the classes parameters is refactored https://review.openstack.org/295309 | 15:15 |
| jeblair | pabelanger: unfortunately, the way we have built the jenkins jobs, the wheel building hostname needs to match; so we will need a new wheel mirror build slave. | 15:15 |
| fungi | anteaya: figuring out why snmp was reporting strange and memory usage on centos 7 inconsistent with what the command line tools reported | 15:15 |
| *** knikolla_ has joined #openstack-infra | 15:15 | |
| pabelanger | jeblair: Ya, just looking at JJB now | 15:16 |
| jeblair | pabelanger: oh wait, i think i'm wrong about that last part, hang oh. | 15:16 |
| fungi | s/and // | 15:16 |
| eharney | thanks yolanda and anteaya | 15:16 |
| yolanda | glad to help | 15:16 |
| anteaya | fungi: ah thank you, sorry I just caught part of it and couldn't figure out the whole context, thanks | 15:16 |
| jeblair | pabelanger: yeah, only the volume name is affected, the node name is different. sorry about that. | 15:17 |
| pabelanger | jeblair: which would you prefer cent-7x64 or centosx64? | 15:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Always do network interface introspection https://review.openstack.org/296450 | 15:18 |
| jeblair | pabelanger: cent7x64 ? | 15:18 |
| pabelanger | WFM | 15:18 |
| anteaya | I missed that we went from 5 to 8 git servers | 15:18 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-openstackid: Order of the classes parameters is refactored https://review.openstack.org/295320 | 15:18 |
| jeblair | fungi: which could mean network problems in dfw | 15:20 |
| jeblair | (ie, in front of the git load balancer) | 15:21 |
| fungi | jeblair: yeah, that's my current suspicion | 15:21 |
| openstackgerrit | Merged openstack-infra/puppet-lodgeit: Order of the classes parameters is refactored https://review.openstack.org/295177 | 15:22 |
| openstackgerrit | Merged openstack-infra/puppet-logrotate: Order of the class parameters is refactored https://review.openstack.org/295179 | 15:23 |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Order of the classes parameters is refactored https://review.openstack.org/295180 | 15:23 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Order of the classes parameters is refactored https://review.openstack.org/295182 | 15:23 |
| openstackgerrit | Merged openstack-infra/puppet-mysql_backup: Order of the classes parameters is refactored https://review.openstack.org/295283 | 15:23 |
| jeblair | oh, we're reordering parameters? | 15:23 |
| *** EricGonczer_ has quit IRC | 15:24 | |
| anteaya | does anyone else feel that perhaps rearranging puppet things on modules before summit is perhaps not the best idea? | 15:24 |
| *** EricGonczer_ has joined #openstack-infra | 15:24 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for httpretty liberty unicode bug 1560808 https://review.openstack.org/296510 | 15:25 |
| openstack | bug 1560808 in OpenStack-Gate "stable/liberty: gate-requirements-integration-dsvm fails with due to UnicodeDecodeError in httpretty" [Undecided,Confirmed] https://launchpad.net/bugs/1560808 | 15:25 |
| jeblair | i feel like there are more substantial things that both authors and reviewers could be spending their time on. | 15:25 |
| *** knikolla_ has quit IRC | 15:25 | |
| anteaya | okay fair point | 15:25 |
| anteaya | I'm in the camp that we change these modules so rarely, yet likely will have to restart services on short notice perhaps doing things that may introduce a bug is not such a good idea | 15:26 |
| jeblair | i'm not going to say anything more about it, because it's also not worth discussing. | 15:26 |
| rossella_s | fungi jeblair can you have a quick look at https://review.openstack.org/#/c/284284 please? I think the only solution is to grant permissions to refs/meta/dashboards , since the dashboard is supposed to be updated daily | 15:26 |
| jeblair | anteaya: i didn't mean to cut off the conversation with you; i typed that last at the same time you were typing :) | 15:27 |
| *** jaosorior has quit IRC | 15:27 | |
| pabelanger | I was the one who approved them. They were in fact just reorders, which no changes to defaults. It was also to clear out the review queue for puppet modules. And don't have a problem reverting them if people feel strongly about it. I tried not to merge anything that would cause rebasing | 15:27 |
| AJaeger | rossella_s: did you see my comments on patch set 4? | 15:27 |
| mordred | pabelanger: I think it's more that it's sad that any of your bandwidth was taken up by the existence of the patches in the first place | 15:27 |
| jeblair | mordred: yeah, that. it's not worth reverting. | 15:28 |
| mordred | pabelanger: we only have one of you, after all - my attempts at cloning you while you sleep notwithstanding | 15:28 |
| *** xiangxinyong has quit IRC | 15:28 | |
| * mordred tells all the clones to hide before someone notices | 15:28 | |
| *** jsavak has quit IRC | 15:28 | |
| *** gyee has joined #openstack-infra | 15:29 | |
| rossella_s | AJaeger, yes Andreas, thanks for that...you said to check on the channel, that's what I am trying to do :) | 15:29 |
| AJaeger | rossella_s: the general direction should work, but it needs proper setup, so enablement of some magic. And fungi and jeblair might be best to help with that | 15:29 |
| AJaeger | rossella_s: ;) | 15:29 |
| fungi | i have a feeling one of the reasons these git remote operations are hitting our repos so hard is that our puppet apply jobs are still cloning over the network rather than starting from the local cache | 15:29 |
| *** x00350071 has joined #openstack-infra | 15:29 | |
| rcarrillocruz | folks | 15:29 |
| anteaya | jeblair: no that's fine, I agree it has little value in discussing | 15:29 |
| rcarrillocruz | in the various clouds infra use | 15:29 |
| *** jsavak has joined #openstack-infra | 15:29 | |
| rcarrillocruz | there's an infra domain (at least in some of them) | 15:29 |
| pabelanger | mordred: right. I only do a few each day. Since my eyes glaze over | 15:30 |
| rcarrillocruz | do you have a domain admin on it? | 15:30 |
| fungi | for example, here's one stuck now for 10 minutes cloning https://git.openstack.org/openstack/puppet-glance | 15:30 |
| jeblair | rcarrillocruz: the only cloud we have any admin on right now is bluebox | 15:30 |
| rossella_s | AJaeger, thanks for all your help...I hope to get this dashboard working :) | 15:30 |
| anteaya | fungi: ah | 15:30 |
| rcarrillocruz | and you just use the cloud admin , yeah? | 15:30 |
| rcarrillocruz | you don't create a specific 'infra' domain admin | 15:30 |
| rcarrillocruz | do you | 15:30 |
| rcarrillocruz | ? | 15:30 |
| pabelanger | jeblair: so, I have run the fs mkmount command, get the following back: fs: You can not change a backup or readonly volume. Where do I look now to ensure it was actually made? My local fs doesn't create the directory | 15:31 |
| *** Qiming has quit IRC | 15:31 | |
| pabelanger | jeblair: I also haven't created the readonly volumes for mirror.wheel.cent7x64 yet | 15:32 |
| mordred | rcarrillocruz: our bluebox cloud isn't really doing v3 | 15:32 |
| rcarrillocruz | ok, thanks | 15:32 |
| rcarrillocruz | then i guess it's not even worth automating user roles and grants in ansible for now, you just use whatever you are given | 15:33 |
| * rcarrillocruz goes back to ansibling | 15:33 | |
| jeblair | pabelanger: aha! afs story time! | 15:34 |
| openstackgerrit | Merged openstack-infra/storyboard: Add API functionality for automatic worklist filters https://review.openstack.org/295422 | 15:35 |
| jeblair | pabelanger: so you grok how afs volumes are mounted into other volumes, so a mountpoint appears as a directory entry? | 15:35 |
| *** tongli has quit IRC | 15:36 | |
| *** nmagnezi has quit IRC | 15:36 | |
| pabelanger | jeblair: not really honestly | 15:36 |
| pabelanger | I'd have to read more up on it | 15:36 |
| jeblair | pabelanger: afs volumes are mounted into other volumes, so a mountpoint appears as a directory entry! :) | 15:36 |
| jeblair | pabelanger: as you move through the afs directory hierarchy, if any of those directories are mountpoints, you will transition from one volume to another... | 15:36 |
| jeblair | pabelanger: /afs/openstack.org/mirror is a volume | 15:37 |
| jeblair | pabelanger: it has several volumes mounted under it (git, npm, pypi, ubuntu)... | 15:37 |
| *** yamahata has quit IRC | 15:38 | |
| pabelanger | ok | 15:38 |
| *** bpokorny has joined #openstack-infra | 15:38 | |
| jeblair | pabelanger: 'fs examine <path>' will tell you about the volume at that path | 15:38 |
| krotscheck | jroll: Not openstack release integrated - (NPM Projects) - there's eslint-config-openstack and the eslint toolchain that run in node, and there's soon to be yeoman-generator-openstack. | 15:39 |
| *** asettle has joined #openstack-infra | 15:39 | |
| pabelanger | jeblair: right, working | 15:39 |
| jeblair | pabelanger: basically the thing to know is that unlike in unix where you mount filesystems on top of each other in kernel memory, and the filessytems don't know about the mountpoints... | 15:40 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 15:40 |
| jeblair | pabelanger: in afs, volume mountpoints are actual filesystem nodes | 15:40 |
| *** jsavak has quit IRC | 15:41 | |
| jeblair | pabelanger: so if you list a directory, some of the things you see might be files, some might be directories, and some might be mountpoints that point to other volumes | 15:41 |
| pabelanger | jeblair: so, does that me, my directory of /afs/openstack.org/mirror/wheel/centos-7-x86_64 should be a directory before issueing the fs mkmount command? | 15:41 |
| *** dizquierdo_afk has quit IRC | 15:41 | |
| pabelanger | or does fs mkmount actually create it? | 15:41 |
| jeblair | pabelanger: nope; fs mkmount will create the mountpoint, and then it will appear as a directory. | 15:42 |
| jeblair | pabelanger: but we're not there yet | 15:42 |
| pabelanger | okay, so missing a step then | 15:42 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-zuul: Order of the classes parameters is refactored https://review.openstack.org/295385 | 15:43 |
| *** jsavak has joined #openstack-infra | 15:43 | |
| openstackgerrit | Merged openstack-infra/puppet-openafs: Order of the class parameters is refactored https://review.openstack.org/295288 | 15:43 |
| jeblair | pabelanger: first i just wanted to convey that as you traverse the afs filesystem, you are invisibly moving from one volume to another. afs is stitching all these volumes together behind the scenes to produce the appearance of a single (global) filesystem. | 15:43 |
| openstackgerrit | Merged openstack-infra/system-config: Temporarily disable Puppet for openstackid.org https://review.openstack.org/296456 | 15:43 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 15:44 |
| *** jaosorior has joined #openstack-infra | 15:44 | |
| *** asettle has quit IRC | 15:44 | |
| pabelanger | jeblair: okay | 15:44 |
| jeblair | pabelanger: now, each time you transition from one volume to another when crossing an afs mountpoint, a little calculation happens -- | 15:44 |
| jroll | krotscheck: okay, you should be aware of https://news.ycombinator.com/item?id=11340510 . essentially consider every package here potentially evil: https://gist.githubusercontent.com/azer/db27417ee84b5f34a6ea/raw/50ab7ef26dbde2d4ea52318a3590af78b2a21162/gistfile1.txt | 15:44 |
| *** markus_z_meeting is now known as markus_z | 15:44 | |
| *** Sukhdev has joined #openstack-infra | 15:44 | |
| openstackgerrit | Merged openstack-infra/system-config: Also protected against OS env vars in expand-groups https://review.openstack.org/296128 | 15:44 |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Order of the classes parameters is refactored https://review.openstack.org/295166 | 15:45 |
| krotscheck | jroll: That's since been resolved. | 15:45 |
| jeblair | pabelanger: if the volume you are leaving is a read-only volume, then, when afs looks up the volume you are entering, it will look up the read-only version of that volume first. | 15:45 |
| jroll | krotscheck: they returned all of those namespaces to the original owner? | 15:45 |
| jeblair | pabelanger: if that fails, it will fall back to the read-write volume. | 15:45 |
| pabelanger | jeblair: Ah, I see | 15:45 |
| jeblair | pabelanger: if you are leaving a read-write volume, afs will look up the read-write volume of the one you are entering. | 15:46 |
| krotscheck | jroll: NOt quite, but NPM has seized and un-unpulblished the things that the dev in question unpublished. | 15:46 |
| jroll | krotscheck: ah ok, good to know. thanks | 15:46 |
| krotscheck | jroll: If anything, though, it adds new priority to us creating a static mirror, because that will isolate us from issues like that. | 15:46 |
| jeblair | pabelanger: so basically, if you have a string of volumes that all have read-only versions, as users move down the line, they will continue to prefer the read-only versions. | 15:47 |
| jroll | krotscheck: maybe, I think you still would have mirrored potentially evil packages | 15:47 |
| *** zeih has joined #openstack-infra | 15:47 | |
| jeblair | pabelanger: so for that reason, the volumes at /afs/openstack.org and mirror/ and pypi/ all have read-only replicas | 15:47 |
| krotscheck | jroll: Yes, but the only mirror software out there for npm doesn't delete old packages, therefore the left-pad package that caused all the issues would still be available to us. | 15:48 |
| jroll | krotscheck: yep | 15:48 |
| jeblair | pabelanger: and *that* is why the fs mkmount errored -- because you can't alter the read-only volume you were on | 15:48 |
| pabelanger | jeblair: Ah, now that make sense | 15:48 |
| krotscheck | jroll: I have to say though, who needs to pull in an external dependency for padding strings? | 15:48 |
| jeblair | pabelanger: this brings up the question "well, how do you actually get to the read-write version of the mirror volume in order to make the new mountpoint?" | 15:48 |
| jroll | krotscheck: yeah, that's the real question here :D | 15:49 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: kilo: cap diskimage-builder<1.1.2 https://review.openstack.org/296555 | 15:49 |
| jroll | krotscheck: though... https://github.com/openstack/oslo.utils/blob/master/oslo_utils/uuidutils.py#L25 :D | 15:49 |
| jeblair | pabelanger: for convenience, there is a convention of creating a forced read-write mountpoint for the cell's root volume in the afs root volume. that is mounted at '/afs/.openstack.org' (note the leading dot before openstack) | 15:49 |
| *** woodster_ has joined #openstack-infra | 15:49 | |
| jeblair | pabelanger: if you cd into that, and then walk down the hierarchy, you will be in the read-write versions of all of those volumes | 15:50 |
| pabelanger | jeblair: gotcha, and there now | 15:50 |
| jeblair | pabelanger: so the really short answer to your question is "make the mountpoint in /afs/.openstack.org/mirror/wheel/" :) | 15:50 |
| pleia2 | good morning | 15:51 |
| jeblair | pleia2: howdy | 15:51 |
| mordred | krotscheck: I, for one, look forward to our npm mirror overlords | 15:51 |
| pabelanger | jeblair: So, looking at http://docs.openstack.org/infra/system-config/afs.html#creating-a-volume the fs mkmount would never work, because we are using /afs/openstack.org right? | 15:51 |
| *** rockyg has joined #openstack-infra | 15:51 | |
| pabelanger | and they are all read-only mounts | 15:51 |
| fungi | we have a _bunch_ of zombie/defunct ansible-playbook processes on the puppetmaster. is that normal? | 15:52 |
| jeblair | pabelanger: in practice, probably not. it *would* work if one of those path components either didn't have a read-only volume, or had a mountpoint that forced the read-write version. we don't generally have that in practice, so yeah, updating that to be '.openstack.org' would probably be the thing to do. | 15:52 |
| anteaya | good morning pleia2 | 15:53 |
| pabelanger | jeblair: and yes, using the updated fs mkmount, my directory now exists | 15:53 |
| jeblair | pabelanger: cool. and you should see it on /afs/.openstack.org/mirror/wheel but not in /afs/openstack.org/mirror/wheel | 15:53 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-refstack: Order of the classes parameters is refactored https://review.openstack.org/295351 | 15:53 |
| SpamapS | jeblair: so I think I've found an oddity/bug/notsure in gertty when you have two gerrit's defined in your .gertty.yaml .. | 15:53 |
| pabelanger | jeblair: correct | 15:53 |
| *** fawadkhaliq has quit IRC | 15:54 | |
| anteaya | fungi: I actually don't know what is normal for processes on the puppetmaster | 15:54 |
| SpamapS | jeblair: it happily allows one to share a database in this case.. | 15:54 |
| pabelanger | jeblair: okay, have to stop here for 30mins, need to walk daughter down to preshool | 15:54 |
| pabelanger | will be back shortly | 15:54 |
| jeblair | pabelanger: because /afs/.openstack.org/mirror is a volume named 'mirror' and /afs/openstack.org/mirror is a volume named 'mirror.readonly' | 15:54 |
| jeblair | pabelanger: cool | 15:54 |
| SpamapS | jeblair: but after doing so, reviewing changes often crashes with this: | 15:54 |
| SpamapS | sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one() | 15:54 |
| SpamapS | jeblair: I suspect that there is need for a 'server' field somewhere and a filter on it. Thoughts? | 15:55 |
| SpamapS | File "/home/clint/venv/gertty/local/lib/python2.7/site-packages/gertty/db.py", line 968, in getAccountByUsername | 15:55 |
| SpamapS | return self.session().query(Account).filter_by(username=username).one() | 15:55 |
| jeblair | SpamapS: yeah, multiple gerrits can not share a database, so if you want to do that, you'll need to specify an alternate dburi for the second server | 15:55 |
| SpamapS | I suspect that place is the table queried by getAccountByUsername. ;) | 15:55 |
| jeblair | SpamapS: probably it would be nice for gertty to check that and say "hey, it looks like you have two servers using the same database uri; don't do that". | 15:56 |
| SpamapS | to be clear, adding a second server without specifying a dburi, results in a corrupted database. | 15:56 |
| SpamapS | so, I think it's a bit more serious than that. | 15:56 |
| fungi | anteaya: well, they're pretty recent so i have a feeling this is a normal part of an ansible run | 15:56 |
| jeblair | SpamapS: (ultimately, it might be nice to alter the schema to support having them both in the same database, and even putting them both in the same ui) | 15:56 |
| SpamapS | (I call it corrupted, because it crashes gertty) | 15:56 |
| *** tongli has joined #openstack-infra | 15:56 | |
| *** roxanaghe has joined #openstack-infra | 15:56 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 15:56 |
| jeblair | SpamapS: yes, corrupted would be a good description. i think it is unrecoverable. | 15:57 |
| anteaya | fungi: okay | 15:57 |
| SpamapS | So an easy way to check this would simply be to refuse to create the row that leads to this .one() fail | 15:57 |
| *** jsavak has quit IRC | 15:57 | |
| *** sridhar_ram1 has joined #openstack-infra | 15:57 | |
| SpamapS | a unique key is likely in order | 15:58 |
| *** jsavak has joined #openstack-infra | 15:58 | |
| *** jamesmcarthur has quit IRC | 15:58 | |
| jeblair | SpamapS: that would work too. we should probably do both things -- sanity check the yaml config, and also the own-user entry. | 15:58 |
| *** apoorvad has joined #openstack-infra | 15:58 | |
| anteaya | I'm going to have lunch, clarkb when you are about ttx is waiting to hear if you want to add another job to https://review.openstack.org/#/c/295499/ | 15:58 |
| jeblair | SpamapS: sorry you got bit by my undocumented assumption. :( | 15:58 |
| openstackgerrit | Merged openstack-infra/puppet-kerberos: Order of the classes parameters is refactored https://review.openstack.org/295172 | 15:58 |
| *** tosky has quit IRC | 15:59 | |
| *** IlyaG has joined #openstack-infra | 15:59 | |
| SpamapS | jeblair: no problem at all, I'm just making sure I understand the assumption so I can patch around it. :) | 15:59 |
| *** apoorvad has quit IRC | 15:59 | |
| *** rkukura has joined #openstack-infra | 15:59 | |
| *** jamesmcarthur has joined #openstack-infra | 15:59 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-storyboard: Order of the classes parameters is refactored https://review.openstack.org/295372 | 15:59 |
| *** apoorvad has joined #openstack-infra | 15:59 | |
| SpamapS | and actually I think I can just delete the extra record in account | 16:00 |
| *** vgridnev has quit IRC | 16:00 | |
| *** rkukura has quit IRC | 16:00 | |
| *** rkukura has joined #openstack-infra | 16:00 | |
| *** yamahata has joined #openstack-infra | 16:00 | |
| jeblair | SpamapS: oh, yeah, if it broke early enough it didn't add any more data than that, that should be easy | 16:00 |
| HeOS | yolanda, hello! I've updated my requests with negative scores. Could you repeat review? :) | 16:01 |
| jeblair | SpamapS: but if it started storing projects and changes... ugh. | 16:01 |
| *** tongli has quit IRC | 16:01 | |
| SpamapS | oh yeah it ran for like a day | 16:01 |
| * SpamapS torches | 16:01 | |
| *** roxanaghe has quit IRC | 16:02 | |
| *** tonytan4ever has quit IRC | 16:02 | |
| *** tonytan4ever has joined #openstack-infra | 16:02 | |
| SpamapS | well at least I can save the subscribed project names and fix that with a quick update on the new db | 16:02 |
| jeblair | SpamapS: yeah, that, and if you've done any project-topics, are the main bits of local-only data | 16:03 |
| SpamapS | the main one is actually the reviewed flag :-P | 16:04 |
| krotscheck | jroll: ya know, UUID management I can accept as a dependency, as there's an actual set of specs surrounding that standard. | 16:04 |
| *** jaypipes has quit IRC | 16:04 | |
| jeblair | SpamapS: that too :| you could probably save that too | 16:04 |
| jeblair | SpamapS: just re-set those after it has re-synced | 16:05 |
| *** esikachev has quit IRC | 16:05 | |
| clarkb | anteaya: ttx why don't we do that in a follow up change since this change is already quite big | 16:05 |
| SpamapS | yeah, I'm awking up the output of selects into updates now ;) | 16:05 |
| *** links has joined #openstack-infra | 16:06 | |
| sdague | hmmm... so fedora guests are only running tempest in 2 workers? | 16:06 |
| *** jamesmcarthur has quit IRC | 16:06 | |
| clarkb | sdague: thats determined by number of cpus on the guest not the distro | 16:06 |
| clarkb | at least it was when I last looked and made tempest run serially | 16:07 |
| *** jaypipes has joined #openstack-infra | 16:07 | |
| sdague | clarkb: https://jenkins05.openstack.org/job/gate-tempest-dsvm-platform-fedora23-nv/35/consoleFull | 16:07 |
| sdague | it's only 2 workers | 16:07 |
| SpamapS | oh hrm, unique keys are already everywhere.. would take major surgery to make it use compound unique keys | 16:07 |
| sdague | the fedora23 jobs are regularly getting to 3 hours and timing out | 16:07 |
| SpamapS | might as well do a server column if we do that | 16:07 |
| sdague | which is putting a big drag on devstack patches in check | 16:07 |
| *** erlon has joined #openstack-infra | 16:08 | |
| fungi | sdague: seems reasonable to switch them to experimental while someone with a vested interest in f23 looks into the issue | 16:08 |
| clarkb | sdague: interesting, maybe fedora has a broken or missing num cpu command thing | 16:08 |
| clarkb | 2 is the default too iirc | 16:08 |
| *** matrohon has quit IRC | 16:09 | |
| clarkb | `nproc` | 16:09 |
| jeblair | clarkb, sdague: that host has 4 processors | 16:09 |
| clarkb | jeblair: that'll do it | 16:09 |
| sdague | jeblair: hmmm... when did we start getting 4 proc nodes? | 16:09 |
| jeblair | sdague: it's news to me :) | 16:10 |
| *** sshnaidm has quit IRC | 16:10 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: New External Orders Endpoints https://review.openstack.org/296574 | 16:10 |
| *** esikachev has joined #openstack-infra | 16:10 | |
| jeblair | i'm poking in nova | 16:10 |
| sdague | jeblair: cool, thanks | 16:10 |
| clarkb | we have been using 4 proc nodes on osic because they are plenty quick, but I don't think w ehave changed anything to make nodepool boot 4 cpu nodes in rax | 16:10 |
| *** zeih has quit IRC | 16:10 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: New External Orders Endpoints https://review.openstack.org/296574 | 16:10 |
| jeblair | | flavor | 8GB Standard Instance (6) | | 16:11 |
| sdague | also, do we have a wheel mirror for fedora? | 16:11 |
| clarkb | dstufft: we do not | 16:11 |
| clarkb | er sdague ^ | 16:11 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Default to hostname not fqdn for laravel env https://review.openstack.org/296453 | 16:11 |
| sdague | ok, that would explain why we're spending a ton of time in pip | 16:11 |
| jeblair | we're using a standard rather than performance flavor for fedora23 | 16:11 |
| mordred | jeblair: that's weird | 16:11 |
| sdague | jeblair: oops, yeh, that would do it | 16:11 |
| jeblair | maybe missed something in nodepool conf | 16:12 |
| mordred | sdague: pabelanger is working on getting centos going in AFS - so I imagine fedora could be next on the list | 16:12 |
| jeblair | yep, patch incoming | 16:12 |
| *** zeih has joined #openstack-infra | 16:13 | |
| sdague | mordred: ok, cool. this fedora run spend 13 minutes building pip packages http://logs.openstack.org/69/295769/3/check/gate-tempest-dsvm-platform-fedora23-nv/d04d885/logs/devstacklog.txt.gz#_2016-03-23_11_01_22_038 | 16:13 |
| jeblair | wait, i'm assuming this is an error, but what if it isn't? | 16:14 |
| jeblair | mordred, clarkb, sdague: is there a chance that ianw or pabelanger ran into problems with either hvm or performance nodes? | 16:14 |
| mordred | jeblair: anything is possible | 16:14 |
| sdague | jeblair: is there any git history about the original change | 16:14 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add docs publishing job to osprofiler https://review.openstack.org/296584 | 16:14 |
| jeblair | sdague: i'm looking, but it's *absence* of things which is harder :) | 16:14 |
| mordred | jeblair: I recommend we don't land a fix until pabelanger gets back online | 16:14 |
| clarkb | jeblair: my guess is no, most of the problems with rax aren't related to that difference, but it is possible | 16:14 |
| jeblair | mordred: agreed. i will write the fix, but we'll get pabelanger/ianw signoff before landing. | 16:15 |
| clarkb | most of it has been systemd + glean | 16:15 |
| * mordred worships ianw for diving in on the systemd + glean stuff | 16:15 | |
| Zara | (hm, I'm concerned that by accepting these reordering patches, to avoid losing time to it, we encourage people to make more, and we lose more time overall.) | 16:16 |
| ttx | clarkb: ack | 16:16 |
| *** HeOS has quit IRC | 16:17 | |
| *** tongli has joined #openstack-infra | 16:17 | |
| clarkb | ttx: responded to https://review.openstack.org/#/c/295500/1 as well | 16:18 |
| krotscheck | mordred: So do I (look forward to the day when I have the time to build that mirror) | 16:18 |
| mordred | Zara: I thnk you are not the only one with such a worry | 16:18 |
| * timrc heard people are gathered around a camp fire here while jeblair talks about openafs | 16:18 | |
| mordred | timrc: yah man - there's some good stuff in the scrollback | 16:19 |
| fungi | smarcet: i've confirmed puppet is being successfully skipped on openstackid.org now (last run hit it at 15:56 utc, current run started at 16:00 and completed by 16:14). the hostname change has merged now and should get applied on openstackid-dev during the current 16:15 run so in a few minutes i expect the service to break there and then i will apply the manual correction to /etc/hostname and make | 16:19 |
| fungi | sure it comes back correctly | 16:19 |
| clarkb | ttx: I will write the change for the stable constraints proposal in a few | 16:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Use correct flavors for centos7 and fedora23 in rax https://review.openstack.org/296585 | 16:19 |
| jeblair | sdague, mordred, clarkb: ^ | 16:19 |
| clarkb | AJaeger: are you good with https://review.openstack.org/#/c/295499/ and its interactions iwth the db changes you are making? its ok for me to approve that now? | 16:19 |
| smarcet | fungi: roger that | 16:19 |
| mattt | anteaya: figured it out, CI instances do not have secure_path set in /etc/sudoers, where-as on stock trusty this is set | 16:19 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: move fedora23 devstack to experimental https://review.openstack.org/296586 | 16:20 |
| sdague | jeblair: cool | 16:20 |
| mattt | anteaya: so locally a sudo was getting the path defined by secure_path, where-as in the gate it was getting the PATH of the calling user | 16:20 |
| sdague | in the mean time https://review.openstack.org/296586 | 16:20 |
| ttx | clarkb: ok, I'll need to check if we have stable branches for all the things using the stable-compat template or not | 16:20 |
| mattt | not sure if you guys want to note that for anyone else who runs into this issue | 16:20 |
| *** jamesmcarthur has joined #openstack-infra | 16:20 | |
| sdague | with a few identified issues to be looked into to get the timing under control | 16:20 |
| mattt | cuz this took a while to troubleshoot :( | 16:20 |
| jeblair | timrc: yes, but pabelanger went out to get more marshmallows :) | 16:20 |
| clarkb | ttx: ok, my guess is we do, thats mostly python libs and oslo | 16:20 |
| *** Guest60842 has quit IRC | 16:21 | |
| sdague | clarkb: also, the rax networking issue still seems to be a thing - http://status.openstack.org//elastic-recheck/#1531187 | 16:21 |
| ttx | anteaya: all clear on https://review.openstack.org/#/c/295499 | 16:21 |
| clarkb | sdague: yes I don't think dib has merged the fix yet :/ | 16:21 |
| sdague | should we take rax out of multinode pool until their maintenance window is done | 16:21 |
| clarkb | sdague: we can take it out while we wait for dib | 16:22 |
| *** tongli has quit IRC | 16:22 | |
| sdague | cool, the current behavior is kind of killing us | 16:22 |
| *** andymaier has quit IRC | 16:22 | |
| fungi | smarcet: success! openstackid is now broken as expected on https://openstackid-dev.openstack.org/ | 16:24 |
| sdague | I'm still trying to land the .gitreview update for stable/mitaka devstack | 16:24 |
| fungi | manually updating the server's hostname to not be the fqdn now | 16:24 |
| sdague | on the 3rd or 4th recheck now | 16:24 |
| clarkb | the fix has merged I guess now we just need a release, image rebuild, and rax uploads to work | 16:24 |
| mordred | jeblair, fungi: ^^ see note from mattt above. it seems weird to me that sudoers would be different in those differnet places | 16:26 |
| fungi | smarcet: manually adjusting /etc/hostname and also ensuring the short hostname appeared as an alias for all versions of the fqdn in /etc/hosts and updating with the hostname command (so that we didn't need a reboot) has corrected the service on openstackid-dev now | 16:26 |
| mordred | mattt: when you say "stock trusty" - what do you mean? | 16:26 |
| mattt | mordred: well if i boot an instance on rackspace cloud my /etc/sudoers looks different | 16:26 |
| mattt | mordred: and by different i mean it has secure_path listed | 16:26 |
| pabelanger | and back | 16:26 |
| mordred | ah | 16:27 |
| mordred | that's not stock | 16:27 |
| pabelanger | just catching up on backscroll | 16:27 |
| mattt | mordred: i have an old 12.04 ubuntu instance on digital ocean and that also has secure_path set | 16:27 |
| mordred | kk | 16:27 |
| mordred | I'll look in to the differences | 16:27 |
| smarcet | fungi: ok | 16:27 |
| *** MarkAtwood has joined #openstack-infra | 16:27 | |
| Zara | mordred: is there anything planned to make sure that can't happen? I've already lost time to review more interesting things today, because I don't want to give it +1 until I have time to look closely, in case it turns out to miss a vital parameter or something, but I'm wary of leaving it too long in case others +1 just to get it over with. | 16:27 |
| mattt | mordred: yeah, that threw us for a loop because when we ran ansible through tox it was installing packages into tox's venv | 16:27 |
| mattt | mordred: and we could not replicate this locally | 16:27 |
| *** skolekonov is now known as _skolekonov | 16:27 | |
| fungi | smarcet: next i'll quickly make those edits and also manually correct the environment.php file for laravel on openstackid.org | 16:27 |
| *** tosky has joined #openstack-infra | 16:28 | |
| pabelanger | jeblair: I am not sure why the wrong flavor is used currently. I don't think we had any issue with performance nodes | 16:28 |
| Zara | mordred: a one-off ain't so bad, but would rather it not be a pattern... | 16:28 |
| mattt | mordred: we forced ansible to use tox's python interpreter so we could run end-to-end in the gate, but it still meant we couldn't run the same test locally | 16:28 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 16:29 | |
| sdague | also, anyone know why logs.openstack.org is now serving at about 300kbs ? (19s for a 772K transfer) | 16:30 |
| fungi | smarcet: here goes... | 16:30 |
| sdague | that's quite a bit slower than it used to be | 16:30 |
| *** tqtran has joined #openstack-infra | 16:30 | |
| reed | pleia2, that storyboard sends emails is worth a # success | 16:30 |
| reed | :) | 16:30 |
| fungi | smarcet: https://openstackid.org/ seems to still be working. can you double-check? | 16:30 |
| fungi | if so i think we're safe to reenable puppet for it | 16:30 |
| pleia2 | reed: I agree! | 16:31 |
| *** nadya has quit IRC | 16:31 | |
| smarcet | fungi: working | 16:31 |
| *** fawadkhaliq has joined #openstack-infra | 16:31 | |
| pleia2 | Zara: care to do a #success for storyboard emails getting to us now? :) | 16:31 |
| Zara | hehehe, sure! | 16:31 |
| clarkb | sdague: is that file coming from swift? | 16:31 |
| *** salv-orlando has joined #openstack-infra | 16:31 | |
| Zara | done! | 16:31 |
| sdague | http://logs.openstack.org/52/294252/8/check/gate-grenade-dsvm/a585f93/logs/grenade.sh.txt.gz#_2016-03-23_14_23_28_233 | 16:31 |
| sdague | don't know | 16:31 |
| *** rajinir has joined #openstack-infra | 16:32 | |
| Zara | I kept typing it in allcaps at first | 16:32 |
| pleia2 | lol | 16:32 |
| *** gordc has joined #openstack-infra | 16:32 | |
| *** tongli has joined #openstack-infra | 16:32 | |
| sdague | I just noticed logs getting a lot slower to pull of late | 16:32 |
| fungi | smarcet: okay, i've approved 296462 to reenable puppet for openstackid.org. as soon as i see a puppet run apply on it successfully and not break the service, i'll launch the new (and hopefully final) 15gb replacement server and we can kick the tires briefly before updating dns to use it as the new production server | 16:32 |
| *** MarkAtwood has quit IRC | 16:32 | |
| smarcet | fungi: cool roger that sire! | 16:33 |
| *** david-lyle has quit IRC | 16:33 | |
| *** tongli has quit IRC | 16:33 | |
| *** tongli has joined #openstack-infra | 16:33 | |
| *** tongli has quit IRC | 16:33 | |
| *** david-lyle has joined #openstack-infra | 16:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Move stable compat jobs up to mitaka https://review.openstack.org/295500 | 16:34 |
| timrc | jeblair: I seriously think we need to make "story time with jeblair" a weekly thing :P You're great at explaining Things. | 16:34 |
| mordred | mattt: oh. yeah. well, that would be because we install our own sudoers files there | 16:34 |
| pleia2 | timrc: I too enjoyed the backlog | 16:34 |
| clarkb | sdague: I don't think any of the grenade logs should be in swift so that woulnd't be it | 16:34 |
| *** scheuran has quit IRC | 16:34 | |
| *** pvaneck has joined #openstack-infra | 16:35 | |
| *** eil397 has joined #openstack-infra | 16:35 | |
| *** fawadkhaliq has quit IRC | 16:35 | |
| *** fawadkhaliq has joined #openstack-infra | 16:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-sudoers: Add secure_path setting to match upstream sudoers https://review.openstack.org/296598 | 16:37 |
| clarkb | ttx: see I97f0cf78bf2c980628abe0f18aa9f5e96af70ae7 apparently we don't want stable constraints updates? thats :( | 16:37 |
| *** zz_dimtruck is now known as dimtruck | 16:37 | |
| *** zeih has quit IRC | 16:37 | |
| *** hichihara has quit IRC | 16:37 | |
| *** ddieterly is now known as ddieterly[away] | 16:38 | |
| mordred | infra-root: ^^ that patch is related to the thing that mattt found above - namely that sudo is configured differently on our nodes than in the wild | 16:38 |
| *** salv-orlando has quit IRC | 16:38 | |
| mattt | mordred: does that not run the risk of breaking a ton of jobs ? | 16:38 |
| ttx | clarkb: ah. hm. Maybe dhellmann can provide context | 16:38 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Temporarily disable Puppet for openstackid.org" https://review.openstack.org/296462 | 16:39 |
| anteaya | ttx: thank you | 16:39 |
| clarkb | ttx: which explains why I didn't find it the first time around | 16:39 |
| mordred | mattt: it certainly does | 16:40 |
| *** abregman has quit IRC | 16:40 | |
| mattt | mordred: doesn't look right to me | 16:40 |
| mordred | mattt: but - that's an example of a file we write out for our sysadmin purposes on our long-lived servers leaking on to our test nodes in a way that might cause actual behavioral differences - so it's certainly worth us talking about what we want to happen | 16:40 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Updated OAUTH2.0 consent UI https://review.openstack.org/296602 | 16:41 |
| anteaya | mattt: thanks for sharing your experience, it is appreciated | 16:41 |
| *** mikelk has quit IRC | 16:41 | |
| mordred | mattt: that's a secure_path I got from a non-puppeted trusty node - what does yours look like? | 16:41 |
| mordred | mattt: oh - wait | 16:41 |
| mordred | heh | 16:41 |
| mattt | mordred: :) | 16:41 |
| anteaya | timrc: ++ to storytime with jeblair | 16:41 |
| mattt | mordred: on mine: Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" | 16:41 |
| *** ddieterly[away] is now known as ddieterly | 16:41 | |
| *** salv-orlando has joined #openstack-infra | 16:42 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-sudoers: Add secure_path setting to match upstream sudoers https://review.openstack.org/296598 | 16:42 |
| mordred | mattt: how about now ? | 16:42 |
| mattt | mordred: yep ! | 16:42 |
| anteaya | Zara: I think HeOS_ is the person doing all the puppet reconfig | 16:42 |
| clarkb | mordred: are those tabs important? I always have to visudo to figure any of that out | 16:43 |
| anteaya | HeOS_: are you aware that one consequence of your patches is causes stress to folks already overworked on reviews? | 16:43 |
| *** yaume_ has quit IRC | 16:43 | |
| anteaya | clarkb: any objection to me +A'ing 295499? | 16:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-sudoers: Add secure_path setting to match upstream sudoers https://review.openstack.org/296598 | 16:44 |
| clarkb | anteaya: nope I think we are good to go | 16:44 |
| mordred | clarkb: good question | 16:44 |
| anteaya | clarkb: thanks | 16:44 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Update Urls for redirects and broken https://review.openstack.org/271243 | 16:44 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Pass through env proxy variables for docs-linkcheck https://review.openstack.org/271244 | 16:44 |
| *** Swami has quit IRC | 16:44 | |
| pabelanger | jeblair: okay, so ready to continue with readonly volumes. I've believe this is the correct syntax to use: vos addsite -server afs01.ord.openstack.org -partition /vicepa -id mirror.wheel.cent7x64 -roid mirror.wheel.cent7x64.readonly | 16:47 |
| pabelanger | jeblair: which shows server afs01.ord.openstack.org partition /vicepa RO Site -- Not released in vos listvldb | 16:48 |
| jeblair | pabelanger: drop the '-roid mirror.wheel.cent7x64.readonly' arguments | 16:48 |
| jeblair | pabelanger: that's optional and it does it for you automatically | 16:48 |
| pabelanger | jeblair: okay | 16:48 |
| mattt | mordred: thanks for your help there btw | 16:48 |
| pabelanger | jeblair: if I've already passed it, do I need to delete the replicated site? | 16:49 |
| pabelanger | or is it just a duplicate setting | 16:49 |
| *** links has quit IRC | 16:49 | |
| jeblair | pabelanger: no, should be okay as long as you didn't typo it :) | 16:49 |
| *** rkukura has quit IRC | 16:49 | |
| jeblair | pabelanger: once it's created, you can 'vos release mirror.wheel.cent7x64' to kick off initial replication | 16:50 |
| *** ifarkas has quit IRC | 16:50 | |
| pabelanger | Released volume mirror.wheel.cent7x64 successfully | 16:50 |
| pabelanger | \o/ | 16:50 |
| pabelanger | okay, so I think everything is setup properly now | 16:51 |
| pabelanger | let me update the JJB template for the new volume name | 16:52 |
| mordred | I do not see anything in /afs/openstack.org/mirror/wheel/ | 16:52 |
| mordred | jeblair: ^^ | 16:52 |
| jeblair | pabelanger: one more thing... | 16:52 |
| jeblair | mordred: yes that's the one more thing :) | 16:53 |
| mordred | :) | 16:53 |
| jeblair | pabelanger: you will need to release the mirror volume in order for your new mountpoint to show up in the read-only path there | 16:53 |
| *** vincentll has quit IRC | 16:53 | |
| *** smarcet has quit IRC | 16:53 | |
| jeblair | i mean the volume named simply 'mirror' | 16:54 |
| jeblair | which is what is mounted at /afs/openstack.org/mirror/ | 16:54 |
| pabelanger | Ah okay | 16:54 |
| *** salv-orlando has quit IRC | 16:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Add mitaka stable branch jobs https://review.openstack.org/295499 | 16:54 |
| *** Guest63327 has quit IRC | 16:55 | |
| *** jerryz has joined #openstack-infra | 16:55 | |
| *** Swami has joined #openstack-infra | 16:55 | |
| pabelanger | done | 16:55 |
| pabelanger | and I can see it on the afsdb01.o.o now | 16:55 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Stop using rax for multinode https://review.openstack.org/296610 | 16:55 |
| clarkb | sdague: ^ | 16:56 |
| *** vgridnev has joined #openstack-infra | 16:56 | |
| *** esikachev has quit IRC | 16:56 | |
| *** cody-somerville has quit IRC | 16:56 | |
| *** cody-somerville has joined #openstack-infra | 16:57 | |
| sdague | fungi: ... I was going to say :) | 16:57 |
| *** tongli has joined #openstack-infra | 16:57 | |
| jeblair | pabelanger: looks good | 16:57 |
| fungi | sdague: i like to exercise my -2 hammer whenever possible, donchaknow? | 16:57 |
| sdague | heh | 16:58 |
| *** EricGonczer_ has quit IRC | 16:58 | |
| *** achanda has joined #openstack-infra | 16:58 | |
| mordred | jeblair, pabelanger: I now see the centos wheel dir | 16:58 |
| mordred | however | 16:58 |
| mordred | ls: cannot open directory /afs/openstack.org/mirror/wheel/centos-7-x84_64/: Permission denied | 16:58 |
| mordred | ls /afs/openstack.org/mirror/wheel/ubuntu-14.04-x86_64/ | 16:59 |
| mordred | works fine | 16:59 |
| jeblair | indeed. it needs some permissions! | 16:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Update centos7 wheel mirror naming schema https://review.openstack.org/296612 | 16:59 |
| jeblair | pabelanger: observe "fs la /afs/openstack.org/mirror/wheel/ubuntu-14.04-x86_64/" | 17:00 |
| jeblair | pabelanger: compared to "fs la /afs/openstack.org/mirror/wheel/centos-7-x84_64/" | 17:00 |
| jeblair | ('la' is short for 'listacl') | 17:00 |
| *** krtaylor has quit IRC | 17:01 | |
| jeblair | (incidentally, apropos of nothing.... 'sa' is short for 'setacl') | 17:01 |
| *** annegentle has joined #openstack-infra | 17:01 | |
| *** tonytan_brb has joined #openstack-infra | 17:01 | |
| mordred | not of nothing - that command is about to be used | 17:01 |
| pabelanger | jeblair: okay, they are different | 17:01 |
| *** tongli has quit IRC | 17:01 | |
| *** burgerk has joined #openstack-infra | 17:01 | |
| jeblair | oh, but mind your read-only / read-write volumes... so you might just want to go ahead and sit yourself in /afs/.openstack.org/mirror/wheel/ | 17:02 |
| *** pots has joined #openstack-infra | 17:02 | |
| *** pvaneck has quit IRC | 17:02 | |
| pabelanger | right, so I need to use fs setacl I suspect? | 17:03 |
| jeblair | pabelanger: yep. | 17:03 |
| *** rossella_s has quit IRC | 17:03 | |
| jeblair | pabelanger: all of those are groups. system:anyuser is special -- it includes all users as well as unauthenticated access | 17:03 |
| *** achanda has quit IRC | 17:03 | |
| *** pvaneck has joined #openstack-infra | 17:04 | |
| *** tonytan4ever has quit IRC | 17:04 | |
| *** rossella_s has joined #openstack-infra | 17:04 | |
| *** jlanoux has quit IRC | 17:04 | |
| *** MarkAtwood has joined #openstack-infra | 17:04 | |
| jeblair | pabelanger: the letters are access rights -- Read, List, Insert, Delete, Write, locK, Administrate | 17:04 |
| mordred | afs acls are the best | 17:05 |
| *** dims_ has quit IRC | 17:05 | |
| jeblair | pabelanger: acls are set on directories (not files), and newly directories inherit the acls of their parets. so this is the perfect time to set those, right before something starts creating entries in a new volume. :) | 17:06 |
| *** zeih has joined #openstack-infra | 17:06 | |
| jeblair | er, "newly created directories" | 17:06 |
| *** sfinucan has quit IRC | 17:06 | |
| *** dims has joined #openstack-infra | 17:06 | |
| pabelanger | okay, I think I did it | 17:07 |
| *** rockyg has quit IRC | 17:07 | |
| jeblair | pabelanger: lgtm; vos release and that should make mordred happy | 17:07 |
| *** Rockyg has joined #openstack-infra | 17:07 | |
| *** ajmiller has quit IRC | 17:07 | |
| mordred | yup | 17:08 |
| mordred | I can now ls and fs la that dir | 17:08 |
| pabelanger | sweet | 17:08 |
| *** ajmiller has joined #openstack-infra | 17:08 | |
| jeblair | and http://mirror.dfw.rax.openstack.org/wheel/centos-7-x84_64/ works | 17:08 |
| mordred | yup. | 17:09 |
| pabelanger | so, 296612 should fix our JJB to use the new volume name | 17:09 |
| mordred | as does http://mirror.iad.rax.openstack.org/wheel/centos-7-x84_64/ | 17:09 |
| mordred | because FAS | 17:09 |
| mordred | because AFS | 17:09 |
| * clarkb is going to have to figure all of this out for the ceph mirror as soon as he figures out how big the ceph mirror needs to be | 17:09 | |
| *** zeih has quit IRC | 17:10 | |
| *** zeih has joined #openstack-infra | 17:10 | |
| jeblair | clarkb: you make a good point | 17:10 |
| *** Rocky_g has joined #openstack-infra | 17:10 | |
| *** rkukura has joined #openstack-infra | 17:10 | |
| jeblair | pabelanger: there's one more one more thing to do :) | 17:10 |
| jeblair | pabelanger: set the quota for the volume | 17:10 |
| *** Rockyg has quit IRC | 17:10 | |
| *** Rocky_g has quit IRC | 17:10 | |
| mordred | clarkb: I was just thinking, in fact, that we should make a nice ansible playboot to set up a new mirror | 17:10 |
| jeblair | pabelanger: (volumes have disk size quotas in afs) | 17:10 |
| *** Rockyg has joined #openstack-infra | 17:11 | |
| fungi | clarkb: one thing we haven't figured out yet is whether we're going to support pre-caching of packages from unofficial package repositories (even ones we're mirroring) and if so what that implementation would actually look like | 17:11 |
| jeblair | mordred: a playboot is the term for a playbook you use to set something up the first time? | 17:11 |
| pabelanger | jeblair: right. I can see there is a difference between the 2 | 17:11 |
| pabelanger | let me ready man for the syntax | 17:11 |
| mordred | jeblair: nope. it's just the name for a playbook that has a bunch of spelling syntax errors | 17:11 |
| clarkb | fungi: pre caching? | 17:12 |
| fungi | clarkb: onto worker images | 17:12 |
| clarkb | fungi: ah yup | 17:12 |
| jeblair | pabelanger: the units are, of course, kilobytes, because we're in the 1980s. | 17:12 |
| *** krtaylor has joined #openstack-infra | 17:12 | |
| fungi | i think building up apt/yum caches locally for packages which aren't in our default sources will get tricky | 17:13 |
| clarkb | fungi: I think that the mirrors are probably suffiicent for these use cases since they aren't super common, however for certain software it may become more ocmmon | 17:13 |
| fungi | so probably better to assume we won't for the time being | 17:13 |
| fungi | yep | 17:13 |
| mordred | we might want to just ride it for a while | 17:13 |
| jeblair | so the default initial quota is 5000 kilobytes, or 5 megabytes. :) | 17:13 |
| mordred | and figure it out when it's an issue | 17:13 |
| mordred | yah | 17:13 |
| pabelanger | jeblair: right. okay, quota updated, and vos release | 17:13 |
| pabelanger | both look the same now | 17:13 |
| jeblair | pabelanger: i don't think you need to vos release for a quota change -- you're really only changing the read-write volume anyway. | 17:14 |
| *** achanda has joined #openstack-infra | 17:14 | |
| jeblair | doesn't hurt anything though | 17:14 |
| pabelanger | okay | 17:14 |
| pabelanger | exciting! Once JJB lands, we can try a build | 17:14 |
| clarkb | is there raeally no good way to find out the size of an apt repo without downloading it all first? | 17:15 |
| pabelanger | this is actually good, and will help with the RPM mirror next | 17:15 |
| *** hashar has quit IRC | 17:15 | |
| *** sshnaidm has joined #openstack-infra | 17:16 | |
| *** ihrachys has quit IRC | 17:16 | |
| jeblair | clarkb: we can adjust the quota at any time, so you can make a guess and go ahead and build it in afs and adjust later. | 17:17 |
| fungi | mmm, 5mb default quota. that's so a couple decades ago | 17:17 |
| mordred | jeblair: ++ | 17:17 |
| clarkb | jeblair: I don't have to worry about the underlying data volumes? | 17:17 |
| mordred | clarkb: nope | 17:17 |
| *** krtaylor has quit IRC | 17:17 | |
| jeblair | clarkb: well, yes, you do have to not exceed the underlying capacity. but we're at 35% on a 1TB volume right now i think. | 17:18 |
| mordred | this is why ceph-rbd backed AFS is such a cool thing | 17:18 |
| fungi | mordred: even with the recent patches merged and applied on the puppetmaster, my launch-node.py run ended in plenty of "/etc/ansible/hosts/openstack:43: Expected key=value host variable assignment, got: argparse" errors | 17:18 |
| jeblair | mordred: yeah, then you don't have to worry about that nearly as much. :) | 17:19 |
| clarkb | mordred: you don't really need AFS with ceph if you just nfs the cephfs | 17:19 |
| clarkb | >_> | 17:19 |
| mordred | clarkb: that's the funniest thing you've ever said | 17:19 |
| mordred | see, it was funny, beause you treated both nfs and cephfs as things someone would use in production | 17:19 |
| mordred | which is clearly irony | 17:19 |
| clarkb | mordred: I mean for mirrors cephfs should be fine | 17:19 |
| mordred | nope | 17:19 |
| jeblair | clarkb: and you can just rsync the nfs data locally instead of using the afs cache :) | 17:19 |
| clarkb | because you can always rebuild from scratch | 17:19 |
| mordred | sure. but there's no atomic release | 17:20 |
| *** tonytan_brb has quit IRC | 17:20 | |
| clarkb | but it looks like cephfs does its own protocol for mounting I thought it nfs'd for some reason | 17:20 |
| mordred | also - cephfs is still "experimental" | 17:20 |
| clarkb | maybe it does that too | 17:20 |
| *** tonytan4ever has joined #openstack-infra | 17:20 | |
| clarkb | mordred: atomic release would just be an index update | 17:20 |
| mordred | so using rbd which is solid, behind afs, which is also solid, seems like a much saner less rube-goldberg approach to me | 17:20 |
| clarkb | mordred: since everyone would be talking to the same backend | 17:20 |
| jeblair | we may also be handwaving around authentication too | 17:21 |
| jeblair | (minor detail) | 17:21 |
| *** mylu has joined #openstack-infra | 17:22 | |
| mordred | fungi: when you have some free cycles, wanna talk about repo signing? | 17:23 |
| *** krtaylor has joined #openstack-infra | 17:23 | |
| clarkb | mordred: the big drawback to doing what you describe is the limit on the number of devices you can attach to hosts so you end up distributing that atop a distributed system | 17:24 |
| *** tphummel has joined #openstack-infra | 17:24 | |
| fungi | mordred: sure, probably free in about an hour if you're around | 17:24 |
| mordred | fungi: cool | 17:24 |
| mordred | clarkb: I may be missing what you're saying - too many pronouns with no antecedants | 17:25 |
| clarkb | mordred: but yes cephfs is still in the probably not great for data you care about phase of its life | 17:25 |
| *** derekh has quit IRC | 17:25 | |
| clarkb | mordred: using rbd you have to attach block devices via iscsi? which runs into the same problem logs.openstack.org has | 17:25 |
| mordred | why would I use iscsi for rbd? | 17:25 |
| mordred | and not an rbd device? | 17:26 |
| SamYaple | mordred: because you hate yourself | 17:26 |
| clarkb | mordred: you end up artificually limited on the number of devices you can use which requires you to build a distributed system atop a distributed system that can halde these problems | 17:26 |
| clarkb | mordred: because cinder | 17:26 |
| mordred | clarkb: we are DEFINITELY talking past each other | 17:26 |
| clarkb | mordred: you could run your own and then not use iscsi, but our current systems use cinder whic is iscsi | 17:26 |
| mordred | bceause that's not what I'm talking about at all | 17:26 |
| clarkb | mordred: its the same problem if you use rbd directly or cinder though | 17:27 |
| *** roxanagh_ has joined #openstack-infra | 17:27 | |
| mordred | clarkb: so - there are two issues: getting disk devices into a ceph cluster, and consuming that ceph cluster as the source of disk in AFS | 17:27 |
| SamYaple | for the record cephfs is not experimental anymore. there are a few 100T+ cephfs pools that i have personally been on | 17:27 |
| mordred | SamYaple: cool. it still has none of the features that I want that AFS has, but good to know that it's out of experimental stage | 17:28 |
| SamYaple | mordred: understood | 17:28 |
| SamYaple | and agreed with | 17:28 |
| clarkb | mordred: yes and if you are limitted to running both in a cloud they have the same challenges and you will be solving them twice | 17:28 |
| mordred | clarkb: ignore the "get disks into ceph cluster" for a sec and assume that a ceph cluster exists that you have access to | 17:28 |
| clarkb | mordred: which is my point, that seems lik ea big operational headache and a drawback | 17:28 |
| mordred | this is not true for us right now, and may never be - but put it to the side for a sec | 17:29 |
| mordred | if you have a ceph cluster that exposes storage via rbd | 17:29 |
| *** rcernin has quit IRC | 17:29 | |
| mordred | then you need one rbd device on the AFS volume server | 17:29 |
| *** achanda has quit IRC | 17:29 | |
| mordred | and the fact that AFS works in quotas rather than volume sizes matches nicely with the expandable amount of storage available on the rbd device | 17:30 |
| * SamYaple is a big ceph and distributed storage guy so his interest is piqued.... | 17:30 | |
| mordred | that's the part of the interfaces matching up that's awesome | 17:30 |
| mordred | but it DOES assume the existence of a ceph cluster | 17:30 |
| clarkb | mordred: and you can arbitrarily grow that beyond 14TB? | 17:30 |
| SamYaple | clarkb: yes | 17:30 |
| mordred | clarkb: remember, we're not talking about managing the ceph cluster | 17:30 |
| clarkb | ok why in the world does no one expose this this way | 17:30 |
| mordred | we're talking about consuming an existing ceph cluster | 17:31 |
| clarkb | because the current thing we have is a giant pain | 17:31 |
| mordred | and we're not talking about consuming ceph via cinder | 17:31 |
| mordred | we're talking about consuming ceph natively | 17:31 |
| *** roxanagh_ has quit IRC | 17:31 | |
| mordred | on the other side, _managing_ the ceph cluster, if you were to do it on top of an openstack cloud that was getting block devices from cinder | 17:31 |
| mordred | then yes, you'd need multiple servers each contributing up to 14TB to the ceph pool | 17:32 |
| mordred | but ceph would happily put those things into the storage pool for you | 17:32 |
| mordred | and you could consume them via rbd in AFS | 17:32 |
| clarkb | sounds like cinder should just wire up rbd directly for us and the world would be magical | 17:32 |
| mordred | all nice and clean like | 17:32 |
| mordred | yah | 17:32 |
| jeblair | (which is basically our strategy with afs right now since we don't have a ceph cluster -- 14TB afs fileservers as needed) | 17:32 |
| mordred | yup | 17:33 |
| fungi | jpmaxman: already mentioned to smarcet elsewhere, but the replacement 15gb openstackid.org is up at 2001:4800:7817:102:be76:4eff:fe05:d9cd and 23.253.97.70 if you want to kick the tires before we change dns to point to it. i've already set the current records down to a 5-minute ttl | 17:33 |
| jeblair | (but yeah, afs as a caching/authz/authn/global-filesystem-view interface to data stored on ceph would be ideal) | 17:33 |
| *** achanda has joined #openstack-infra | 17:33 | |
| mordred | yup. it's one of the best storage system combos evar | 17:33 |
| openstackgerrit | Merged openstack-infra/shade: Always do network interface introspection https://review.openstack.org/296450 | 17:33 |
| mordred | old-school solidity of AFS with new-fangled dynamic sizing of ceph | 17:34 |
| SamYaple | so forgive my coming in late to this conversation, but this RBD that we would use is just going to be attached to a single machine right? we arent talking about using it as a distributed filesystem? | 17:34 |
| mordred | SamYaple: yup | 17:34 |
| *** tlbr has joined #openstack-infra | 17:34 | |
| mordred | SamYaple: to both things | 17:34 |
| SamYaple | cool then AFS would make it distributed | 17:34 |
| mordred | yah | 17:34 |
| SamYaple | should work well | 17:34 |
| mordred | now - for _scaling_ purposes one might want to make multiple rbd devices and attach them to different AFS fileservers | 17:34 |
| SamYaple | you can dynamically grow (and shrink) the rbd | 17:35 |
| mordred | in case the main afs fileserver became a bottleneck | 17:35 |
| mordred | but I'd want to see that happen before I architected for it | 17:35 |
| *** smarcet has joined #openstack-infra | 17:35 | |
| jpmaxman | fungi - great! | 17:35 |
| mordred | given the other levels of caching and whatnot | 17:35 |
| SamYaple | individual rbds also have a performance bottlenext sometimes | 17:35 |
| mordred | SamYaple: exactly | 17:35 |
| clarkb | right I think ^ is why everyone chunks to 1TB everywhere | 17:35 |
| mordred | yah. but that's a next step problem | 17:35 |
| SamYaple | at around the ~500MB/s scale ive had to attach multiple RBDs and raid0 them to get more performance | 17:35 |
| clarkb | because its a general problem in storage land | 17:35 |
| mordred | and one that can be solved _behind_ AFS | 17:36 |
| mordred | without interface issues | 17:36 |
| openstackgerrit | Merged openstack-infra/project-config: move fedora23 devstack to experimental https://review.openstack.org/296586 | 17:36 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: dhcp-all-interfaces depends on dhcp https://review.openstack.org/296626 | 17:36 |
| *** fawadkhaliq has quit IRC | 17:36 | |
| mordred | thing of it like MVC for storage | 17:36 |
| SamYaple | will these AFS servers be using the RBD linux kernel module? or is this qemu using the rbd python libs? | 17:36 |
| mordred | SamYaple: oh - we have no ceph | 17:36 |
| *** fawadkhaliq has joined #openstack-infra | 17:37 | |
| mordred | or plans for a ceph currently | 17:37 |
| mordred | we're just doing AFS on top of dumb cinder block devices right now | 17:37 |
| pleia2 | AJaeger: quick question on 296348 (good change, I'd like to see it land) | 17:37 |
| *** jamesmcarthur has quit IRC | 17:37 | |
| jeblair | if one of our clouds offered ceph rbd as a service, this is how we would use it :) | 17:37 |
| SamYaple | i.... may be confused then. is this a hypotheical in-ten-years conversatin? | 17:37 |
| mordred | SamYaple: no, this is how cern runs their AFS | 17:37 |
| mordred | SamYaple: in a world where I set up a ceph cluster and then an AFS cluster on top of that cluster | 17:37 |
| SamYaple | yea | 17:37 |
| mordred | SamYaple: I would be using the RBD linux kernel module | 17:37 |
| mordred | on the AFS fileserver | 17:38 |
| SamYaple | so this has no practical purpose to implement something today? | 17:38 |
| mordred | SamYaple: right now we're at 35% capacity on a single 1TB volume | 17:38 |
| mordred | it would be a bit overkill for today | 17:38 |
| clarkb | SamYaple: the practical purpose today would be if ^ gre wto 14TB | 17:38 |
| mordred | yah | 17:38 |
| mordred | like, if we replaced static.o.o file storage with AFS filestorage | 17:38 |
| mordred | for instance | 17:38 |
| jeblair | clarkb: well, if we grow to 14tb, we can just add another afs fileserver. | 17:38 |
| mordred | yah | 17:39 |
| jeblair | we *could* run a ceph cluster, but we don't have to. | 17:39 |
| clarkb | jeblair: but you have to shard right? probably fine for our use case but not always the greatest solution | 17:39 |
| *** fawadkhaliq has quit IRC | 17:39 | |
| SamYaple | ok. well should this ever materialize, rbd under AFS is a good plan | 17:39 |
| mordred | SamYaple: ++ | 17:39 |
| *** fawadkhaliq has joined #openstack-infra | 17:39 | |
| SamYaple | and running ceph in vms with cinder volumes backing ceph is also not a BAD plan | 17:39 |
| SamYaple | just fyi | 17:40 |
| SamYaple | ive done that before | 17:40 |
| jeblair | clarkb: we probably don't want giant volumes for other reasons anway -- volume operations like moving, replication, etc, would take a while. | 17:40 |
| mordred | SamYaple: it can't be worse that our current "stich 14 cinder volumes into LVM" | 17:40 |
| clarkb | jeblair: in theory with cephfs you could have 15TB files :) | 17:40 |
| *** jamesmcarthur has joined #openstack-infra | 17:40 | |
| jeblair | we *might* want to more seriously consider backing afs with ceph today if we we want to make our read-write volumes more resilient to failure. | 17:40 |
| clarkb | jeblair: or even $otherfs on rdb | 17:40 |
| SamYaple | well no. the idea is you pull another 1TB from cinder whenever needed and attach it as another disk. then you create another osd in the vm and then you can grow your rbd | 17:40 |
| *** fawadkhaliq has quit IRC | 17:41 | |
| jeblair | clarkb: okay. i don't want a 15tb file. | 17:41 |
| jeblair | i think that's too big, even for subunit. | 17:41 |
| *** achanda has quit IRC | 17:41 | |
| *** fawadkhaliq has joined #openstack-infra | 17:41 | |
| clarkb | how many hours of 4k video is that | 17:41 |
| *** rbrndt has quit IRC | 17:42 | |
| jeblair | it's more than a library of congress | 17:42 |
| fungi | jpmaxman: smarcet: i've changed the dns records for openstackid.org from 104.130.154.113 and 2001:4800:7818:103:53f7:aea1:b964:d61c to 23.253.97.70 and 2001:4800:7817:102:be76:4eff:fe05:d9cd now | 17:42 |
| SamYaple | clarkb: about 3 minutes | 17:42 |
| *** ddieterly is now known as ddieterly[away] | 17:42 | |
| clarkb | 60fps 4k video is supposedly ~4TB an hour | 17:43 |
| *** bpokorny has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish tacker content on *.openstack.org https://review.openstack.org/295036 | 17:43 |
| clarkb | so one of the hobbit films is ~20TB? | 17:43 |
| SamYaple | lets ppick something good clarkb | 17:43 |
| smarcet | fungi: roger that | 17:43 |
| mordred | SamYaple: "Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters | 17:43 |
| mordred | " | 17:43 |
| mordred | SamYaple: still experimental by my definition | 17:44 |
| clarkb | SamYaple: well the hobbit shot 2x4k 60fps | 17:44 |
| SamYaple | mordred: that is really old docs, though its still in master i agree | 17:44 |
| Clint | clarkb: you might want to compress it | 17:44 |
| fungi | mordred: that reminds me of the last time i tried to use btrfs ;) | 17:44 |
| SamYaple | mordred: thats been around since.... firefly? | 17:44 |
| mordred | SamYaple: just saying | 17:44 |
| SamYaple | mordred: fungi the btrfs docs still say that too | 17:44 |
| fungi | hah | 17:44 |
| mordred | SamYaple: for good reason | 17:44 |
| SamYaple | though they annouced it as stable over a year ago | 17:44 |
| SamYaple | mordred: youre wrong and should feel bad for spreading FUD | 17:44 |
| clarkb | Clint: yup, I was just trying to wrap my head around the high end | 17:45 |
| mordred | it's not stable until you've removed the "warning, don't use this" from your docs | 17:45 |
| clarkb | so ya 15TB files probably not common | 17:45 |
| SamYaple | mordred: kernel devs modifying docs? pffff | 17:45 |
| bkero | AIUI that's pretty small beans compared to a lot of ceph users | 17:45 |
| *** electrofelix has quit IRC | 17:45 | |
| mordred | mind you - I'm convinced that ceph for block storage is rock solid and scalable and awesome | 17:45 |
| *** jamesmcarthur has quit IRC | 17:45 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Disable tempest in networking-odl rally job. https://review.openstack.org/296637 | 17:45 |
| SamYaple | mordred: im not pushing cephfs, dont worry. i was just commenting on it. | 17:46 |
| bkero | mordred: That warning has been there for a long time. I'm wondering if it's applicable anymore. | 17:46 |
| mordred | SamYaple: totes | 17:46 |
| bkero | Maybe they just have a docs problem :) | 17:46 |
| mordred | bkero: yah - that's my thing ... as a user, i'm going to consider the warning in the official docs valid until removed | 17:46 |
| SamYaple | bkero: its a little of both | 17:46 |
| *** fawadkhaliq has quit IRC | 17:47 | |
| *** yamamoto_ has quit IRC | 17:47 | |
| *** fawadkhaliq has joined #openstack-infra | 17:47 | |
| bkero | mordred: The block storage is also built off of the object storage, so that should be considered rock solid too. Can some workloads be adapted to use that? Or is that the now-you-have-two-problems problem? | 17:48 |
| *** salv-orlando has joined #openstack-infra | 17:48 | |
| *** jamesmcarthur has joined #openstack-infra | 17:48 | |
| SamYaple | you guys are going to love the new bluestore for ceph | 17:48 |
| clarkb | bkero: the problem with object storage that we have found is that you have to build an entire application layer on top of it to essentially replace rbd and a filesystem | 17:48 |
| SamYaple | who needs xfs or know filesystems? | 17:48 |
| clarkb | bkero: becues you need indexing, and you need file lookups, and you need stuff that can grab the objects | 17:49 |
| *** piet has quit IRC | 17:49 | |
| clarkb | bkero: so you can either write that from scratch or take advantage of 20 something years of software development | 17:49 |
| bkero | clarkb: I'm pretty sure all those problems have been solved in libraries | 17:50 |
| fungi | also note that when rewriting posix filesystem semantics from scratch, you're bound to rediscover all the mistakes you predecessors have solved | 17:50 |
| clarkb | bkero: they haven't we know because we have attemted to use swift as a store for logs | 17:50 |
| clarkb | bkero: really the best library for filesytem things is a filesystem | 17:50 |
| bkero | fungi: Yeah, attempting to write a vfs layer on top of object store = s3fs, which has and has worked around a lot of that | 17:50 |
| SamYaple | fungi: of note: CephFS will never be fully, POSIX compatible, they have stated as much | 17:51 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenStackID 1.0.16 https://review.openstack.org/296641 | 17:51 |
| bkero | clarkb: Yup. By adapt workload I mean use !filesystem | 17:51 |
| *** rbrndt has joined #openstack-infra | 17:51 | |
| clarkb | bkero: right but then you find out you need things a filesystem does | 17:51 |
| *** bpokorny has joined #openstack-infra | 17:51 | |
| mordred | bkero: right but we don't _want_ to not use the filesystem | 17:51 |
| *** rbrndt has quit IRC | 17:51 | |
| mordred | the filesytem is not a bug | 17:51 |
| clarkb | bkero: you need indexes and listings and organizational structure | 17:51 |
| mordred | it's a very good piece of software | 17:51 |
| clarkb | mod times | 17:51 |
| mordred | that does very good things | 17:51 |
| *** mriedem1 has joined #openstack-infra | 17:51 | |
| mordred | s/filesystem/object-storage/ is not always a win | 17:51 |
| clarkb | you also tend to want software to be able to talk to the data store | 17:51 |
| *** achanda has joined #openstack-infra | 17:51 | |
| fungi | people wanting to browse a list of files you're serving may want a heirarchy to more easily navigate, timestamping which indicates when things deeper in the heirarchy have changed, et cetera | 17:52 |
| bkero | mordred: Yep. It's just a win for sharding purposes. It's worse in about every other respect for this application. | 17:52 |
| *** mriedem has quit IRC | 17:52 | |
| clarkb | objects stores do make sense for very specific use cases but require work to use | 17:52 |
| *** piet has joined #openstack-infra | 17:52 | |
| *** yamamoto has joined #openstack-infra | 17:52 | |
| *** cdent has quit IRC | 17:53 | |
| mordred | bkero: yup. and the sharding thing can be solved by using layers - such as object-store -> rados -> rbd -> AFS :) | 17:53 |
| bkero | Or any manner of clustering filesystem, sure | 17:53 |
| mordred | yah | 17:53 |
| clarkb | I think really where we tripped up is our use case for logs is very specifically solved by a filesystem | 17:53 |
| mordred | yup | 17:53 |
| mordred | they are files with a directory structure and also will be presented as such | 17:54 |
| clarkb | we need navigable structured indexes with modification time records | 17:54 |
| *** dloi has joined #openstack-infra | 17:54 | |
| fungi | smarcet: jpmaxman: cacti is reflecting the ram increase now http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2067&rra_id=all | 17:54 |
| *** tongli has joined #openstack-infra | 17:54 | |
| *** angdraug has joined #openstack-infra | 17:54 | |
| bkero | clarkb: how does a filesystem address multiple indexes for quickly searching that data in several ways though? | 17:54 |
| *** electrofelix has joined #openstack-infra | 17:54 | |
| mordred | bkero: it doesn't - but we don' thave that | 17:54 |
| mordred | bkero: when we want that - we put files into elasticsearch | 17:54 |
| *** tongli has quit IRC | 17:55 | |
| clarkb | bkero: but also an object store doesn't provide that at all | 17:55 |
| mordred | ya. that too | 17:55 |
| clarkb | bkero: an object store gives you flat lists | 17:55 |
| clarkb | you can have different flat lists but ya | 17:55 |
| *** tongli has joined #openstack-infra | 17:55 | |
| clarkb | at least with swift you can attach metadata and then list all the things matching specific flags | 17:55 |
| clarkb | but they aren't organized in any manner against each other | 17:55 |
| smarcet | fungi: ok, | 17:56 |
| bkero | Nope, but that's useful functionality | 17:56 |
| clarkb | flags is probably the wrong word there, basically you can filter on metadata attributes | 17:56 |
| clarkb | bkero: but its not something we need | 17:56 |
| bkero | Tags might be more familiar | 17:56 |
| smarcet | fungi: now we could stretch legs :) | 17:56 |
| mordred | yah. I mean, object storage is cool | 17:57 |
| mordred | and does many cool things | 17:57 |
| *** jamesmcarthur has quit IRC | 17:57 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: OpenStackID 1.0.16 https://review.openstack.org/296641 | 17:57 |
| mordred | it just happens that those things do not actually match up with our actual problem space currently | 17:57 |
| fungi | updated ^ to link to git.o.o instead of github | 17:57 |
| *** jamesmcarthur has joined #openstack-infra | 17:57 | |
| mordred | which is great- luckily the world has more than one tech solution | 17:57 |
| mordred | fungi: is that a think we want to land? | 17:58 |
| smarcet | fungi: ok sorry aout htat | 17:58 |
| tosky | (sorry for the interruption) few days ago I asked how to import a filtered subset of commits from one repository to another | 17:58 |
| tosky | one of the first step is pushing the imported commits for review using a new (orphan) branch (no common history) | 17:59 |
| tosky | now, as I can't create a new branch, the question is: how should I do this? Propose a change for gerrit to temporarily allow it (enabling it for cores)? Or...? | 17:59 |
| *** mriedem has joined #openstack-infra | 18:00 | |
| *** mriedem1 has quit IRC | 18:00 | |
| *** zeih has quit IRC | 18:00 | |
| *** sfinucan has joined #openstack-infra | 18:00 | |
| *** jordanP has quit IRC | 18:00 | |
| anteaya | I have created an etherpad with a draft email to the list about bots: https://etherpad.openstack.org/p/bots-bots-bots | 18:00 |
| anteaya | based on yesterday's discussion | 18:00 |
| *** zeih has joined #openstack-infra | 18:01 | |
| anteaya | I'd be grateful if anyone could see their way clear to spending a moment reviewing | 18:01 |
| fungi | mordred: it's smarcet's change to roll latest openstackid to production. someone with an account on translate-dev should probably test it out just to make sure logins are still working there | 18:01 |
| *** salv-orl_ has joined #openstack-infra | 18:01 | |
| fungi | since it reflects the current state of the deployment on openstackod-dev, which translate-dev is using | 18:01 |
| mordred | fungi: cool. thanks!~ | 18:02 |
| mordred | fungi: I understand the patch now | 18:02 |
| *** jsavak has quit IRC | 18:02 | |
| smarcet | fungi: seems to we working | 18:02 |
| fungi | awesome | 18:02 |
| *** jsavak has joined #openstack-infra | 18:02 | |
| clarkb | tosky: you can push an acl change to add the gerrit create permission | 18:03 |
| *** eil397 has quit IRC | 18:03 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: update solum to use the correct release jobs https://review.openstack.org/296647 | 18:03 |
| tosky | clarkb: so that's the way - but who should be the target? Can I do it for a specific user? All the current configuration is about groups | 18:03 |
| clarkb | tosky: you can only do it for groups in gerrit iirc. Use the release/core group | 18:04 |
| clarkb | tosky: the smaller the group the better | 18:04 |
| *** salv-orlando has quit IRC | 18:04 | |
| tosky | clarkb: yes, that was the question (how to avoid a -2 even before the CI ends :) | 18:04 |
| clarkb | load average: 32.19, 32.79, 35.01 probably plays a part in slow log downloads | 18:05 |
| openstackgerrit | Merged openstack-infra/openstackid: Updated OAUTH2.0 consent UI https://review.openstack.org/296602 | 18:05 |
| *** Sukhdev has quit IRC | 18:06 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:06 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=137&rra_id=all started a couple weeks ago | 18:06 |
| *** yuanying_ has joined #openstack-infra | 18:07 | |
| *** lucasagomes is now known as lucas-dinner | 18:08 | |
| fungi | smarcet: jpmaxman: i've got a packet capture going on the old openstackid.org watching for new incoming https connections. still getting a few but it looks like the majority are landing at the new server now | 18:08 |
| *** vgridnev has quit IRC | 18:08 | |
| *** MarkAtwood has quit IRC | 18:08 | |
| *** achanda has quit IRC | 18:09 | |
| jpmaxman | woot! | 18:09 |
| smarcet | fungi: ok | 18:09 |
| jpmaxman | thanks | 18:09 |
| smarcet | fungi: thx a lot ! | 18:09 |
| fungi | most recent was 3 minutes ago | 18:09 |
| *** yuanying has quit IRC | 18:09 | |
| smarcet | fungi: yes i was performing some massive request to server :P | 18:09 |
| openstackgerrit | Merged openstack-infra/shade: Add environment_files to stack_create https://review.openstack.org/277559 | 18:09 |
| clarkb | new logstash pipeline possibly more performant? Or the added load for downloading all of those subunit files? | 18:10 |
| *** piet has quit IRC | 18:10 | |
| jpmaxman | smarcet: looking good in my tests | 18:11 |
| fungi | clarkb: ouch, that's severe | 18:11 |
| fungi | clarkb: i wouldn't expect better-performing logstash to cause this, since it would just reduce the latency for its retrievals not the number of them | 18:12 |
| clarkb | total bits out eth0 http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all doesn't look that much worse | 18:12 |
| clarkb | fungi: ya and I don't think its added subunit downloads because ^ doesn't show a jump | 18:12 |
| fungi | clarkb: looks like it's definitely apache though. i'll see what looks hardest hit | 18:12 |
| *** EricGonczer_ has joined #openstack-infra | 18:13 | |
| anteaya | cj: I commented on your gerritbot patch | 18:13 |
| *** jaosorior has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack-infra/system-config: Use environment file in sample reverse DNS command https://review.openstack.org/296494 | 18:13 |
| *** kzaitsev_mb has joined #openstack-infra | 18:13 | |
| clarkb | total numbers of tcp connections don't seem to have changed dramatically either | 18:14 |
| *** annegentle has quit IRC | 18:14 | |
| *** krotscheck has quit IRC | 18:14 | |
| *** krotscheck has joined #openstack-infra | 18:15 | |
| AJaeger | pleia2: I answered on 296348 - I think we do not use automatic labels anywhere. But if you feel strongly, I can change... | 18:15 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Fix devstack config for devstack-plugin-nfs jobs https://review.openstack.org/296657 | 18:15 |
| *** ericksonsantos has quit IRC | 18:16 | |
| fungi | clarkb: wow, lots of 404s being reported for logs.o.o requests | 18:17 |
| pleia2 | AJaeger: not worth the trouble, just seemed like duplication of effort since we already have a link | 18:17 |
| clarkb | fungi: yes beacuse it doesn't actually know what files exist for which jobs | 18:17 |
| clarkb | fungi: so thats normal | 18:17 |
| *** mrmartin has quit IRC | 18:17 | |
| pleia2 | AJaeger: anyway, approved | 18:17 |
| *** jamesmcarthur has quit IRC | 18:17 | |
| fungi | yeah, those are logstash workers groping around, confirmed | 18:17 |
| clarkb | fungi: that should be cheap | 18:18 |
| clarkb | but maybe it isn't | 18:18 |
| fungi | clarkb: well, they go through a cgi | 18:18 |
| openstackgerrit | Merged openstack-infra/project-config: Stop using rax for multinode https://review.openstack.org/296610 | 18:18 |
| *** MarkAtwood has joined #openstack-infra | 18:19 | |
| *** annegentle has joined #openstack-infra | 18:19 | |
| fungi | clarkb: did we add a bunch of new filenames to check for a week or two ago? | 18:19 |
| clarkb | fungi: not that I know of | 18:19 |
| clarkb | but it is possible | 18:19 |
| *** cdent has joined #openstack-infra | 18:19 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Enhance zuul setup section https://review.openstack.org/296348 | 18:19 |
| *** yamamoto has quit IRC | 18:19 | |
| *** nmagnezi has joined #openstack-infra | 18:20 | |
| clarkb | fungi: its also possible the subunit side chagned | 18:20 |
| *** vgridnev has joined #openstack-infra | 18:20 | |
| clarkb | there are two of them now | 18:21 |
| fungi | the only access logs getting updated at all quickly are logs, releases and tarballs fwiw | 18:21 |
| clarkb | huh whats hitting tarballs? | 18:21 |
| *** amuller has joined #openstack-infra | 18:21 | |
| amuller | Good afternoon | 18:21 |
| fungi | clarkb: jobs that pip install things like the neutron master branch tarball because stadium or something | 18:22 |
| clarkb | fungi: gotcha | 18:22 |
| clarkb | fungi: so the "simple" test here is to just turn off half the logstash workers and see if load falls by half | 18:22 |
| pabelanger | fungi: clarkb: pleia2 do you mind helping land https://review.openstack.org/#/c/296612/ ? And update for our wheel.mirror centos volume | 18:23 |
| *** esker has joined #openstack-infra | 18:23 | |
| AJaeger | thanks for reviewing, pleia2 | 18:23 |
| amuller | We started seeing failures in neutron functional job because rabbit wouldn't start: http://logs.openstack.org/42/296542/1/check/gate-neutron-dsvm-functional/4764cd9/console.html#_2016-03-23_16_23_40_001 | 18:23 |
| *** jsavak has quit IRC | 18:24 | |
| *** yamamoto has joined #openstack-infra | 18:24 | |
| fungi | amuller: started when? | 18:24 |
| amuller | not sure, I noticed 2 such failures today | 18:24 |
| pabelanger | amuller: look like a package issue from the logs | 18:24 |
| craige | o/ | 18:24 |
| *** jsavak has joined #openstack-infra | 18:24 | |
| smarcet | fungi: i think that we would need to tweak apache conf | 18:24 |
| amuller | let's try logstash | 18:24 |
| pabelanger | wonder if there is a new release or something | 18:24 |
| smarcet | i am getting now on load tests connection refused | 18:24 |
| amuller | I wanted to ask if you guys saw this anywhere else | 18:24 |
| *** zeih has quit IRC | 18:24 | |
| smarcet | with 500 concurrent users | 18:25 |
| amuller | It doesn't happen every time though (That'd be too simple) | 18:25 |
| clarkb | EmilienM had rabbit issues and the RDO packages conflicting over uids, but thta wasn't on ubuntu | 18:25 |
| fungi | smarcet: agreed, that seems like the next thing to try increasing is apache worker/thread levels | 18:25 |
| AJaeger | clarkb, fungi, pabelanger: I have a question on https://review.openstack.org/294224 - I suggest to use other-requirements.txt instead of a specific builder for installation of elasticsearch. But can bindep handle install of "https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.5.deb" ? | 18:25 |
| clarkb | fungi: smarcet you can use etherpad's apache conf as a template for that | 18:25 |
| clarkb | AJaeger: I don't think it can | 18:26 |
| fungi | smarcet: we may also hit max open files limits if we try to turn it up too far, and may need to increase that as well | 18:26 |
| *** ericksonsantos has joined #openstack-infra | 18:26 | |
| clarkb | AJaeger: if the apt repo for elastisearch is added before hand then it should work though | 18:26 |
| *** ihrachys has joined #openstack-infra | 18:26 | |
| *** krotscheck has quit IRC | 18:26 | |
| *** krotscheck has joined #openstack-infra | 18:26 | |
| pabelanger | clarkb: AJaeger: if the repo is added I think other-requirements.txt will be fine | 18:27 |
| pabelanger | right | 18:27 |
| EmilienM | clarkb: what is context? sorry | 18:27 |
| clarkb | amuller: I would try installing rabbitmq on ubuntu-trusty to see if you can reproduce | 18:27 |
| clarkb | EmilienM: rabbitmq install failing at http://logs.openstack.org/42/296542/1/check/gate-neutron-dsvm-functional/4764cd9/console.html#_2016-03-23_16_23_40_001 | 18:27 |
| AJaeger | clarkb, pabelanger so, how can we add the repo? | 18:27 |
| clarkb | amuller: you might also be able to check syslog | 18:27 |
| EmilienM | clarkb: afik gate-neutron-dsvm-functional does not deploy openstack with packaging. | 18:27 |
| amuller | clarkb: rabbit itself has a log: http://logs.openstack.org/42/296542/1/check/gate-neutron-dsvm-functional/4764cd9/logs/rabbitmq/startup_log | 18:28 |
| clarkb | syslog has nothing logged | 18:28 |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| clarkb | amuller: well there you go :) | 18:28 |
| *** piet has joined #openstack-infra | 18:28 | |
| clarkb | EmilienM: and it runs on ubuntu | 18:28 |
| clarkb | EmilienM: I do not think they are related issues | 18:28 |
| smarcet | fungi: ok i could write a patch for puppet conf to tweak apache | 18:28 |
| *** jamesmcarthur has quit IRC | 18:28 | |
| *** jsavak has quit IRC | 18:28 | |
| *** mestery_ has joined #openstack-infra | 18:28 | |
| EmilienM | clarkb: we use rabbitmq repos (which are often down) but at least we have recent releases | 18:28 |
| smarcet | fungi: max open files is from sys ctl rite ? | 18:29 |
| pabelanger | AJaeger: they'd have to do the following first: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html | 18:29 |
| AJaeger | pabelanger: and that needs to be done as builder I guess, correct? | 18:29 |
| amuller | My Google Fu lead me to some pages saying that you need to add the host's hostname to /etc/hosts to avoid that rabbitmq error | 18:29 |
| amuller | is it possible that some cloud provider changed something hostname related? that would explain why this is not happening every time | 18:30 |
| fungi | smarcet: yeah, the apache tuning we do on the etherpad server for example is https://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/files/apache-connection-tuning | 18:30 |
| *** rbrndt_ has joined #openstack-infra | 18:30 | |
| pabelanger | AJaeger: Right. Otherwise, we'd have to setup a mirror for it in AFS | 18:30 |
| AJaeger | pabelanger: thanks, then I'll know how to advise on that change. | 18:31 |
| *** jaypipes has quit IRC | 18:31 | |
| fungi | smarcet: we install it at https://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/manifests/apache.pp#n91 | 18:31 |
| pabelanger | AJaeger: I guess a future enhancement to bindep would be to allow local packages some how | 18:31 |
| pabelanger | maybe fungi has an idea | 18:31 |
| clarkb | amuller: its possible glean isn't properly setting that based on the config drive info | 18:31 |
| *** sambetts is now known as sambetts|afk | 18:31 | |
| clarkb | amuller: I can hop on one built from the same image in the same region and see what it looks like | 18:32 |
| pabelanger | AJaeger: also, 296612 when you get finished your current reviews please | 18:32 |
| mordred | clarkb: one of these days we're going to finally solve hostname for once and for all | 18:32 |
| smarcet | fungi: ok i will take care of that | 18:32 |
| amuller | clarkb++ | 18:33 |
| *** yamamoto has quit IRC | 18:33 | |
| *** mestery has quit IRC | 18:33 | |
| *** mestery_ is now known as mestery | 18:33 | |
| fungi | pabelanger: clarkb: yeah, i'm not sure bindep is a great solution when you need nonstandard package repositories, though you could create additional (non-default_ bindep requirements files for those i suppose | 18:33 |
| *** angdraug has quit IRC | 18:33 | |
| clarkb | amuller: http://paste.openstack.org/show/491625/ is what /etc/hosts looks like | 18:33 |
| cj | thanks, anteaya | 18:33 |
| *** yamamoto has joined #openstack-infra | 18:33 | |
| fungi | you can tell bindep to process a specific list filename rather than the default | 18:33 |
| *** yamamoto has quit IRC | 18:33 | |
| clarkb | I wonder if its binding to the public interface then trying to do a self test which fails on the firewall | 18:34 |
| *** electrofelix has quit IRC | 18:34 | |
| *** yamamoto has joined #openstack-infra | 18:34 | |
| clarkb | mordred: ^ seems like it should be 127.0.0.1 that gets the name? | 18:34 |
| *** mestery_ has joined #openstack-infra | 18:34 | |
| anteaya | cj: welcome, did what I say make sense to you? | 18:34 |
| amuller | clarkb: FWIW I checked dpkg output from a month ago and from today and rabbitmq package is the same | 18:34 |
| *** vgridnev has quit IRC | 18:35 | |
| AJaeger | pabelanger: 612 is approved | 18:35 |
| fungi | smarcet: for the open files limits on our etherpad deployment, we set them in the upstart script for it https://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/templates/upstart.erb but that was for the node.js process it uses rather than for apache | 18:35 |
| mordred | clarkb: we put it on the IP if the host actually knows its ip | 18:35 |
| smarcet | fungi : roger that | 18:35 |
| mordred | clarkb: if the host does not know its ip, we put it on 127.0.1.1 | 18:35 |
| smarcet | thx for all the info, and for your help of course :) | 18:35 |
| clarkb | mordred: ok that seems ot break rabbit :/ | 18:35 |
| fungi | smarcet: not sure at what point it might be likely to impact apache itself (if at all), or if it might be tuning for the php5-fpm processes | 18:35 |
| mordred | clarkb: AWESOME | 18:35 |
| mordred | why would that break rabbit? | 18:36 |
| amuller | clarkb: not sure why I'm seeing this only on the functional job and not tempest jobs for example | 18:36 |
| *** jsavak has joined #openstack-infra | 18:36 | |
| mordred | is there a firewall that's preventing the node from talking to itself on its real interface? | 18:36 |
| clarkb | mordred: I think it is binding to that interface then doing a self test | 18:36 |
| clarkb | mordred: yes | 18:36 |
| smarcet | fungi: fyi i dont think that we are using php5-fpm | 18:36 |
| mordred | oh for the love of all that is ... | 18:36 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add release notes job for Kolla repo https://review.openstack.org/296129 | 18:36 |
| smarcet | fungi: i think that we are using php apache mod on fork mode | 18:36 |
| clarkb | amuller: if I had to guess devstack is configuring rabbit to listen on a specific interface | 18:37 |
| *** lakshmiS has joined #openstack-infra | 18:37 | |
| clarkb | one that functions | 18:37 |
| clarkb | I can dig in devstack relaly quick to see | 18:37 |
| AJaeger | lakshmiS: are you here to discss searchlight? | 18:37 |
| fungi | smarcet: oh, that may just be getting installed as an unnecessary dependency, but it's definitely running. likely not hurting anything | 18:37 |
| *** sridhar_ram1 is now known as sridhar_ram | 18:37 | |
| lakshmiS | AJaeger: yes. i just replied to your comment | 18:37 |
| *** yamamoto has quit IRC | 18:37 | |
| AJaeger | lakshmiS: That's why I asked here ;) | 18:37 |
| mordred | clarkb: we can reverse the glean decision if we need to - although I'd honestly say that if a config is expecting loopback connection to avoid firewall on the public interface that it should configure itself that way | 18:38 |
| mordred | and otherwise it should make sure it can talk to itself on the real interface | 18:38 |
| cody-somerville | jeblair: Hey. Any more thoughts on my gertty changes? :) | 18:38 |
| amuller | clarkb: http://logs.openstack.org/42/296542/1/check/gate-neutron-dsvm-functional/4764cd9/console.html#_2016-03-23_16_21_13_438 | 18:38 |
| AJaeger | lakshmiS: What you need to do is: Take the existing python27 job, copy it, and add a builder that installs the packages. | 18:38 |
| *** jaypipes has joined #openstack-infra | 18:38 | |
| *** krotscheck has quit IRC | 18:38 | |
| mordred | haha | 18:38 |
| mordred | RABBIT_PASSWORD=127.0.0.1 | 18:38 |
| mordred | that's an awesome password | 18:38 |
| *** krotscheck has joined #openstack-infra | 18:39 | |
| clarkb | mordred: I think its common to not set the public interface | 18:39 |
| *** permalac has quit IRC | 18:39 | |
| clarkb | mordred: because relative you to your you address is localhost | 18:39 |
| AJaeger | lakshmiS: Then use that copied python27 job everywhere instead of the original. Best name it something specific to avoid confusion | 18:39 |
| *** rbrndt_ has quit IRC | 18:39 | |
| *** annegentle has quit IRC | 18:39 | |
| clarkb | amuller: ya RABBIT_HOST there may do it | 18:39 |
| mordred | clarkb: relative to me, the address of my hostname is the same | 18:39 |
| mordred | clarkb: and how I get to the thing that the hostname resolves to is a matter for routing | 18:39 |
| *** jamesmcarthur has joined #openstack-infra | 18:40 | |
| pabelanger | AJaeger: thanks | 18:40 |
| AJaeger | lakshmiS: also, as we discussed, you could just add the repository and use bindep to install all the packages using other-requirements.txt in your repo. | 18:40 |
| amuller | clarkb: it's set to 127.0.0.1 in that failed run though | 18:40 |
| clarkb | mordred: yup and 127.0.0.1 is a routing decision | 18:40 |
| clarkb | amuller: it may not be ending up on disk then? | 18:40 |
| morgan | clarkb: can we make 127.0.0.1 publically routable (ok ok, i'll go back under my rock and stop being part of the peanut gallery) | 18:40 |
| AJaeger | lakshmiS: But that could be a second step later as well | 18:40 |
| clarkb | amuller: its definitely not using localhost | 18:40 |
| morgan | :P | 18:40 |
| mordred | clarkb: I mean - 127.0.0.1 has a name, it's localhost. 72.99.116.160 has a name and ubuntu-trusty-rax-dfw-9019582 is it | 18:41 |
| clarkb | mordred: yup and really the test should use localhost | 18:41 |
| mordred | yes | 18:41 |
| mordred | because that's the thing that the test wants | 18:41 |
| mordred | totally agree | 18:41 |
| AJaeger | lakshmiS: do you understand what I suggest? Or shall I take over your change and do it? | 18:41 |
| *** sdake_ has joined #openstack-infra | 18:41 | |
| clarkb | but separately I am trying to figure out what the common behavior for listing hostname in /etc/hosts is | 18:42 |
| lakshmiS | AJaeger: are you suggesting modelling over https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml? | 18:42 |
| cj | anteaya: yep! I'll get that explanation pushed up. I've got quite a few more changes nearly ready to review as well. We've been using the updates internally and they have made it much easier to track down problems. | 18:42 |
| *** vgridnev has joined #openstack-infra | 18:42 | |
| clarkb | and I think it comes down to what reverse lookup is used to determine myself | 18:42 |
| mordred | clarkb: good luck. there seem to be MANY differeing and opposed opinions | 18:42 |
| clarkb | does it check every interface? loopback only? something else entirely? | 18:42 |
| anteaya | cj: who is Andrew? | 18:42 |
| AJaeger | pabelanger, fungi, clarkb: One more question for bindep: Please review https://review.openstack.org/295551 - ianw's devstack install is oddand I'm not sure whether it's great or wrong ;( | 18:43 |
| *** ddieterly[away] has quit IRC | 18:43 | |
| anteaya | cj: awesome, now I understand the context for your patch, context really helps, thank you | 18:43 |
| lakshmiS | AJaeger: if its easier for you to modify it than explaining me, I will take it :) | 18:43 |
| anteaya | cj: does it make sense why we don't hard code things in the gerribot code? | 18:43 |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| *** yamamoto has quit IRC | 18:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Update centos7 wheel mirror naming schema https://review.openstack.org/296612 | 18:44 |
| cj | of, of course. Sorry those made their way in to the repos. | 18:44 |
| *** sdake has quit IRC | 18:44 | |
| AJaeger | lakshmiS: give it a try first. Model it after https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml?#L96 - the gate-{name}-python27 job-template | 18:44 |
| mordred | clarkb: hah. here's one that does both: http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap9sec95.html | 18:44 |
| anteaya | cj: no worries, as long as you understand the reason | 18:44 |
| AJaeger | lakshmiS: but if you're confused, just tell me... | 18:44 |
| anteaya | cj: thanks, and thanks for your patience :) | 18:44 |
| cj | anteaya: Andrew is tykeal, my boss and the head of the release engineering team at the linux foundation | 18:44 |
| amuller | clarkb: something changed yesterday night according to logstash (message:"invoke-rc.d: initscript rabbitmq-server") | 18:44 |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| anteaya | cj: ah okay thanks | 18:45 |
| *** sfinucan has quit IRC | 18:45 | |
| amuller | clarkb: and it's not the rabbitmq deb package version | 18:45 |
| lakshmiS | AJaeger: I wil take another shot at it... | 18:45 |
| *** yamamoto has quit IRC | 18:45 | |
| clarkb | amuller: could be the version of glean on that image | 18:45 |
| clarkb | amuller: since that is what is setting up /etc/hosts | 18:45 |
| clarkb | mordred: yes I think doing both is what I hvae commonly seen | 18:45 |
| cj | anteaya: in any case, those absolute paths you were referring to were part of the comments describing what the config file format looks like | 18:45 |
| clarkb | mordred: that way a reverse lookup on both IPs gives you the correct thing | 18:45 |
| fungi | AJaeger: what's the question on 295551? looks like it has it's own unit testing and requirement installing stuff | 18:45 |
| AJaeger | thanks, lakshmiS | 18:45 |
| clarkb | mordred: it however does not solve the forward lookup situation I don't think | 18:46 |
| amuller | clarkb: Can we determine if the image was changed since last night? | 18:46 |
| clarkb | amuller: yes | 18:46 |
| anteaya | cj: ah sorry, as comments they are fine | 18:46 |
| *** mestery_ has quit IRC | 18:46 | |
| AJaeger | fungi: so, "./tools/install_prereqs.sh" is fine? Or should we ask for bindep usage? | 18:46 |
| *** tojuvone has quit IRC | 18:46 | |
| clarkb | 00:19:08:36 and 03:20:11:19 | 18:46 |
| lakshmiS | AJaeger: thank you. I am sure i will be back with questions. | 18:46 |
| clarkb | so it updated 19 hours ago and previous time before that was almost 4 days ago | 18:47 |
| *** sdake has joined #openstack-infra | 18:47 | |
| *** eil397 has joined #openstack-infra | 18:47 | |
| cj | anteaya: meh, I don't think so. Users shouldn't need to read the code to find example configurations | 18:47 |
| fungi | AJaeger: it's reusing a lot of devstack's internal package tracking and installation routines https://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/install_prereqs.sh | 18:47 |
| AJaeger | fungi, and was that just never needed because of the larger devstack iamges? | 18:47 |
| pabelanger | Question about global-requirements. How do they change on stable branches, if at all? Looking at the centos7 wheel builds, it seems PyECLib 1.7.0 fails on our stable branches, but this was fixed in master 5 weeks ago with https://review.openstack.org/#/c/278945/ . So, would a backport we done to fix wheels on old branches, or do we just live with PyECLib failing to build properly | 18:47 |
| anteaya | cj: either way | 18:47 |
| amuller | clarkb: that'sp robably it then | 18:47 |
| cj | anteaya: alright. I'll get the changes in to a new patchset | 18:48 |
| fungi | AJaeger: a future to do item will probably be to look at converting devstack to use bindep instead of the current scattered distro package tracking it does, but that's a big project to get into right now | 18:48 |
| anteaya | cj: thanks | 18:48 |
| amuller | clarkb: for my own education can I ask how do you check when images are updated? | 18:48 |
| AJaeger | fungi, thanks. Then let me review waht they do and probably +2 the change. | 18:48 |
| anteaya | pabelanger: yes, usually changes to global-requirements master get backported as far back as any changes need to go | 18:48 |
| clarkb | amuller: I had to ask the nodepool service which isn't publicly exposed. There is a change however to publish this info to http://nodepool.openstack.org/ not sure if that has langed | 18:49 |
| clarkb | *landed | 18:49 |
| *** angdraug has joined #openstack-infra | 18:49 | |
| fungi | AJaeger: the commit message isn't clear, but refers to https://review.openstack.org/294191 which is adding dependencies in devstack | 18:49 |
| amuller | clarkb: gotcha, thanks | 18:49 |
| amuller | clarkb: "infra magic" then =D | 18:49 |
| clarkb | amuller: well its supposed to not be magic soon if not already | 18:49 |
| *** sdake_ has quit IRC | 18:49 | |
| *** vgridnev has quit IRC | 18:50 | |
| amuller | clarkb: is it possible that the new image is not used 100% of the time? | 18:50 |
| *** angdraug has quit IRC | 18:50 | |
| clarkb | amuller: each image is updated at a different time in different cloud regions | 18:50 |
| amuller | clarkb: ok, so we're probably going to approach to 100% failure rate soon | 18:51 |
| clarkb | amuller: but once a region is upated all subsequent boots should be off of that image | 18:51 |
| *** openstack has joined #openstack-infra | 19:08 | |
| fungi | collisions | 19:08 |
| *** openstack has joined #openstack-infra | 19:21 | |
| *** teran_ is now known as teran | 19:21 | |
| *** rkukura has quit IRC | 19:22 | |
| mordred | abregman: we use the heck out of ansible-puppet - but it does not have many needs at the moment | 19:22 |
| *** mhayden has joined #openstack-infra | 19:22 | |
| fungi | oh, wow, suddenly pages of irc just came through in my client after several minutes of lag | 19:22 |
| *** vkramskikh has joined #openstack-infra | 19:22 | |
| mordred | abregman: the general topic of working with rcarrillocruz on using ansible to launch nodes is an important ansible-based topic | 19:22 |
| jeblair | pabelanger: yes, the quota is an attribute of the volume, and acls are attributes of the directories inside the volume. the mountpoint is just a pointer (you can even mount volumes multiple places) | 19:22 |
| anteaya | fungi: thanks | 19:23 |
| anteaya | fungi: fun | 19:23 |
| pabelanger | jeblair: ya, I am happy it was an easy fix | 19:23 |
| *** openstackstatus has joined #openstack-infra | 19:23 | |
| *** ChanServ sets mode: +v openstackstatus | 19:23 | |
| *** zhiyan_ is now known as zhiyan | 19:23 | |
| *** serverascode_ is now known as serverascode | 19:23 | |
| fungi | smarcet: i just approved it. i'm still seeing some straggler connections coming through for the old server, but we can probably ignore them as those are likely people on very stale dns caches now | 19:24 |
| anteaya | abregman: so what do you want to do now | 19:24 |
| *** DuncanT_ is now known as DuncanT | 19:24 | |
| fungi | the original records are long past their ttl | 19:24 |
| *** therve has joined #openstack-infra | 19:24 | |
| *** dtantsur has joined #openstack-infra | 19:24 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add docs publishing job to osprofiler https://review.openstack.org/296584 | 19:24 |
| anteaya | abregman: do you get any value from my replies to your paste or no? | 19:24 |
| *** sigmavirus24 has joined #openstack-infra | 19:25 | |
| *** Guest69253 is now known as zeus` | 19:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix devstack config for devstack-plugin-nfs jobs https://review.openstack.org/296657 | 19:25 |
| fungi | looks like our bots are all back again for the moment | 19:25 |
| abregman | anteaya: yes, I am. but I really want to start working on something. I really don't care how small or not important that change would be..I just want to start contributing | 19:26 |
| anteaya | fungi: they don't seem to like meeting-4 | 19:26 |
| abregman | is there a plan maybe to switch from puppet to ansible? :) | 19:26 |
| anteaya | abregman: okay, well I see communicating as contributing, but I can see how you would feel offer patches constitutes as contributing | 19:27 |
| *** mylu has quit IRC | 19:27 | |
| anteaya | abregman: no | 19:27 |
| anteaya | abregman: we have too much investment in our puppet modules | 19:27 |
| anteaya | we do however use ansible heavily for orchestration | 19:27 |
| anteaya | as puppet leads to race conditions often | 19:27 |
| *** sbadia has joined #openstack-infra | 19:28 | |
| anteaya | whereas ansible can understand the concepts of before and after | 19:28 |
| *** alaski_ is now known as alaski | 19:28 | |
| abregman | anteaya: and what exactly do you use for it? I mean is it a specific project or some playbooks on specific servers? | 19:28 |
| anteaya | mordred is by far the best person who understands ansible in this capacity | 19:29 |
| anteaya | I really hope he has a few moments to spend with you some day | 19:29 |
| mordred | abregman: we use ansible to run puppet | 19:29 |
| anteaya | and to be honest, it changes so fast and is so complex I don't fully understand it | 19:29 |
| mordred | as ansible can handle multi-server orchstration and puppet cannot | 19:29 |
| mordred | we also plan to have ansible boot our nodes when we need new ones | 19:30 |
| mordred | so that we can have that be driven from git | 19:30 |
| mordred | as currently it is an admin operation | 19:30 |
| *** JasonF is now known as JayF | 19:30 | |
| mordred | to create a new server | 19:30 |
| *** Hal has joined #openstack-infra | 19:30 | |
| mordred | and there is no need for it to be such | 19:30 |
| *** Hal is now known as Guest65562 | 19:30 | |
| abregman | mordred: cool. I have experience with writing ansible playbooks/roles/templates. is there a project I can help with? | 19:30 |
| mordred | (one sec- on the phone, sorry) | 19:31 |
| openstackgerrit | Merged openstack-infra/system-config: OpenStackID 1.0.16 https://review.openstack.org/296641 | 19:31 |
| fungi | anteaya: openstack joined #openstack-meeting-4 at 19:23 utc | 19:32 |
| anteaya | fungi: yay thank you | 19:32 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 19:32 |
| *** jsavak has quit IRC | 19:33 | |
| *** zeus` is now known as zeus | 19:33 | |
| AJaeger | dougwig: what do you think of this networking-odl job change? Could you review i, please? https://review.openstack.org/#/c/296637/1 | 19:33 |
| anteaya | fungi: chanserv didnt' change the mode, which is what I had seen in other channels, sorry about that | 19:34 |
| *** zeus has quit IRC | 19:34 | |
| *** zeus has joined #openstack-infra | 19:34 | |
| *** jsavak has joined #openstack-infra | 19:34 | |
| *** annegentle has joined #openstack-infra | 19:34 | |
| fungi | anteaya: chanserv may be out to lunch there | 19:34 |
| fungi | or rather have dropped before that /join | 19:34 |
| pabelanger | so close, apparently tempfile not a centos command | 19:34 |
| fungi | anteaya: though it does appear to be an op in that channel | 19:34 |
| anteaya | so it does | 19:35 |
| anteaya | thanks :) | 19:35 |
| *** hashar has joined #openstack-infra | 19:35 | |
| fungi | pabelanger: you probably need mktemp instead? | 19:35 |
| *** achanda has joined #openstack-infra | 19:35 | |
| fungi | pabelanger: if memory serves, tempfile is mostly deprecated in favor of mktemp anyway | 19:36 |
| pabelanger | fungi: yup, that is what I am testing now | 19:36 |
| *** mylu has joined #openstack-infra | 19:37 | |
| *** nadya has joined #openstack-infra | 19:37 | |
| fungi | pabelanger: according to https://bugs.debian.org/627564 Clint has strong feelings on the subject ;) | 19:38 |
| openstack | Debian bug 627564 in debianutils "debianutils: 'tempfile' should be deprecated" [Minor,Fixed] | 19:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to mktemp for wheel-index.sh https://review.openstack.org/296716 | 19:38 |
| pabelanger | fungi: works for me ^ | 19:38 |
| *** Sukhdev has joined #openstack-infra | 19:39 | |
| anteaya | so if a channel op has a moment to reset the topic in -meeting, that would be great | 19:39 |
| anteaya | sorry in -alt | 19:39 |
| fungi | anteaya: yeah, i'll take care of cleaning up any which are idle | 19:39 |
| anteaya | thanks | 19:40 |
| *** rkukura has joined #openstack-infra | 19:40 | |
| fungi | looks like the topics in the others were fine | 19:41 |
| anteaya | great | 19:41 |
| anteaya | -4 never did bother to start a meeting | 19:41 |
| amuller | clarkb: https://review.openstack.org/#/c/296707/, awesome :) | 19:41 |
| anteaya | I told them to copy their backscroll and post to the mailing list | 19:42 |
| anteaya | as their target audience is operators, who I'm not sure would figure out truncated logs | 19:42 |
| Clint | fungi: not strong enough really | 19:42 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight https://review.openstack.org/294224 | 19:42 |
| *** simondodsley has joined #openstack-infra | 19:43 | |
| *** dims_ has joined #openstack-infra | 19:43 | |
| *** dims has quit IRC | 19:45 | |
| anteaya | AJaeger: I'm foggy on the purpose of 294224 | 19:45 |
| *** yamamoto has joined #openstack-infra | 19:45 | |
| anteaya | AJaeger: are we trying to get searchlight to show up in logstash? | 19:46 |
| fungi | anteaya: they want to install elasticsearch to use in their unit tests | 19:46 |
| *** kzaitsev_mb has quit IRC | 19:46 | |
| anteaya | elasticsearch as a part of searchlight? | 19:46 |
| fungi | anteaya: searchlight is a horizon plugin or something which implements fancy searching with an elasticsearch backend | 19:46 |
| *** kzaitsev_mb has joined #openstack-infra | 19:47 | |
| anteaya | oh | 19:47 |
| AJaeger | anteaya: best ask lakshmiS | 19:47 |
| fungi | so, yeah, when you install searchlighht in your horizon-using openstack deployment, you install elasticsearch as a dependency | 19:47 |
| *** nmagnezi has quit IRC | 19:47 | |
| anteaya | that was the part I missed | 19:47 |
| anteaya | fungi: ah okay thanks | 19:47 |
| david-lyle | searchlight is standalone, but provides a horizon plugin as well | 19:47 |
| anteaya | david-lyle: thanks | 19:47 |
| anteaya | was thinking that this was trying to get searchlight test results in logstash | 19:48 |
| anteaya | which had me very confused | 19:48 |
| fungi | got it. i'm mostly parroting probably outdated things i learned at their presentation in vancouver, so no doubt that's changed | 19:48 |
| lakshmiS | thanks david-lyle | 19:48 |
| david-lyle | there is also a osc plugin | 19:48 |
| fungi | neat | 19:48 |
| anteaya | david-lyle lakshmiS can we get a nice clear commit message so that other fools like me don't waste your time with the same question? | 19:48 |
| *** zeih_ has joined #openstack-infra | 19:49 | |
| anteaya | like "this is a backend for searchlight" in there somewhere | 19:49 |
| anteaya | "not an attempt to have our test results in logstash.openstack.org" | 19:49 |
| AJaeger | clarkb's change to switch stable-compat jobs to mitaka is ready IMHO, dims_ confirmed that we have stable/mitaka for all oslo projects. So, IMHO we can merge https://review.openstack.org/#/c/295500/ | 19:50 |
| lakshmiS | anteaya: sure. will do. i dont know much on what it means to be in logstack, so thanks for asking | 19:50 |
| clarkb | AJaeger: we need it for the clinets too I think? | 19:50 |
| clarkb | AJaeger: let me check | 19:50 |
| anteaya | lakshmiS: wonderful | 19:50 |
| clarkb | git review -d doesn't protect against using the wrong project anymore? | 19:51 |
| anteaya | lakshmiS: logstash.openstack.org is our gui for searching though log errors which elastic search gives us for test results | 19:51 |
| AJaeger | clarkb: yes, please double check | 19:51 |
| clarkb | verions 1.25.0 | 19:51 |
| clarkb | AJaeger: its just oslo that I see so yes I think we can approve that if oslo all have mitaka branches | 19:52 |
| *** smarcet has quit IRC | 19:52 | |
| anteaya | clarkb: what do you mean git review -d doesn't proect against using the wrong project, what does it do now if you run that command in the wrong project? | 19:53 |
| AJaeger | clarkb: dims_ confirmed - and I did some quick checks before that - that all oslo rpos have mitaka branches | 19:53 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight https://review.openstack.org/294224 | 19:53 |
| clarkb | anteaya: it appears to have downloaded the other roject for me | 19:53 |
| *** dims_ has quit IRC | 19:53 | |
| anteaya | clarkb: :( | 19:53 |
| clarkb | anteaya: basically it fetched that ref and checked it out | 19:53 |
| *** yamamoto has quit IRC | 19:53 | |
| anteaya | clarkb: that is going to make for some fun support help questions | 19:54 |
| abregman | mordred: I have to run. I'll try to talk with you later. | 19:54 |
| anteaya | abregman: thank you! | 19:54 |
| *** rocky_g has joined #openstack-infra | 19:54 | |
| abregman | no, thank you anteaya :) | 19:54 |
| *** abregman is now known as abregman|afk | 19:54 | |
| anteaya | lakshmiS: thank you, much clearer for me now | 19:54 |
| *** zeih_ has quit IRC | 19:54 | |
| anteaya | I appreciate it | 19:54 |
| lakshmiS | thanks for the note | 19:54 |
| AJaeger | lakshmiS: nearly there - let me do two small changes on it. | 19:55 |
| openstackgerrit | Rob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack https://review.openstack.org/293090 | 19:55 |
| ianw | jeblair: on 296585 (node type for fedora/centos) ; yeah ... that was just an oversight on my behalf | 19:55 |
| *** dims has joined #openstack-infra | 19:56 | |
| anteaya | lakshmiS AJaeger since these are brand new jobs, with a brand new builder, how do we feel about running them in experiemental first? | 19:56 |
| anteaya | perhaps on one repo to get started? | 19:56 |
| dims | AJaeger : thanks | 19:57 |
| AJaeger | anteaya: good idea, let me do that... | 19:57 |
| anteaya | thanks | 19:57 |
| *** smarcet has joined #openstack-infra | 19:58 | |
| *** roxanaghe has joined #openstack-infra | 19:59 | |
| *** ajmiller_ is now known as ajmiller | 20:00 | |
| *** Sukhdev has quit IRC | 20:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Install Elasticsearch for searchlight https://review.openstack.org/294224 | 20:02 |
| AJaeger | lakshmiS, anteaya : Updated. lakshmiS once this is in add a review comment "check experimental" to run the new job | 20:02 |
| openstackgerrit | Jon Schlueter proposed openstack-infra/shade: Use OpenStackCloudException when delte_instance() raises https://review.openstack.org/296731 | 20:03 |
| *** roxanaghe has quit IRC | 20:03 | |
| *** yamahata has quit IRC | 20:04 | |
| *** ihrachys_ has joined #openstack-infra | 20:04 | |
| AJaeger | lakshmiS: your problem : job-groups in jenkins and templates in zuul are two different entities, you needed to do this in both places | 20:04 |
| *** ociuhandu has quit IRC | 20:05 | |
| openstackgerrit | Jon Schlueter proposed openstack-infra/shade: Use OpenStackCloudException when _delete_server() raises https://review.openstack.org/296731 | 20:05 |
| anteaya | AJaeger: thank you, lets give it a spin | 20:05 |
| *** ihrachys has quit IRC | 20:05 | |
| AJaeger | ;) | 20:06 |
| AJaeger | anteaya, could you review https://review.openstack.org/295500 as well, please? See also clarkb's comments above that this can go in... | 20:07 |
| *** zeih_ has joined #openstack-infra | 20:07 | |
| *** shardy_ has joined #openstack-infra | 20:07 | |
| anteaya | AJaeger: I will | 20:07 |
| AJaeger | thanks | 20:07 |
| openstackgerrit | Merged openstack-infra/project-config: Allow test slaves to talk to themselves https://review.openstack.org/296707 | 20:08 |
| *** shardy has quit IRC | 20:08 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 20:09 |
| *** Daisy has joined #openstack-infra | 20:09 | |
| anteaya | AJaeger: I see clarkb's comment in backscroll, approving | 20:10 |
| *** EricGonczer_ has quit IRC | 20:10 | |
| anteaya | that was in reference to 295500 | 20:10 |
| AJaeger | tahnks! | 20:11 |
| AJaeger | thanks ;) | 20:11 |
| AJaeger | ianw: your hexdump change works, the unittest succeeded on https://review.openstack.org/#/c/294191 | 20:11 |
| anteaya | welcome | 20:12 |
| ianw | AJaeger: such a fuss for one little tool :( anyway, we're getting there | 20:13 |
| AJaeger | ianw, one tiny step after the other ;) | 20:13 |
| *** Daisy has quit IRC | 20:13 | |
| *** BigWillie has quit IRC | 20:13 | |
| anteaya | last call for reviews of this draft email to the -dev list about bots | 20:14 |
| anteaya | https://etherpad.openstack.org/p/bots-bots-bots | 20:14 |
| anteaya | I've had no reviews thus far will wait one more time | 20:14 |
| *** jsavak has quit IRC | 20:14 | |
| * AJaeger reviews | 20:14 | |
| anteaya | thank you | 20:15 |
| *** jsavak has joined #openstack-infra | 20:15 | |
| *** rajinir has quit IRC | 20:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable tempest in networking-odl rally job. https://review.openstack.org/296637 | 20:15 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Clarify location of large output log files https://review.openstack.org/296767 | 20:15 |
| lakshmiS | AJaeger: thanks | 20:16 |
| *** e0ne has joined #openstack-infra | 20:17 | |
| AJaeger | anteaya: I made some mior corrections on wording, hope it's not a change to Denglish ;) | 20:18 |
| AJaeger | minor I mean ;( | 20:18 |
| openstackgerrit | Merged openstack-infra/project-config: Move stable compat jobs up to mitaka https://review.openstack.org/295500 | 20:18 |
| *** achanda has quit IRC | 20:18 | |
| anteaya | AJaeger: thank you, looks fine to me | 20:18 |
| AJaeger | anteaya: good to go for me | 20:18 |
| anteaya | AJaeger: I appreciate your review | 20:18 |
| anteaya | :) | 20:18 |
| anteaya | I see pleia2 has the tab open, will make a cup of tea to allow her time to share her thoughts | 20:19 |
| *** openstack has joined #openstack-infra | 20:32 | |
| jroll | mriedem: whee, ty | 20:32 |
| *** Sukhdev has joined #openstack-infra | 20:33 | |
| mriedem | anteaya: did you want to un-+W https://review.openstack.org/#/c/296772/ ? | 20:33 |
| Zara | anteaya: yeah, that'd be the place I'd want to start with it; for now I'm hoping to use the thread as a general overview on bots in openstack, because I only know about meetbot and openstackgerrit (and even then, probably not that much!). so I'm lurking away. :) | 20:33 |
| anteaya | mriedem: I have done so | 20:33 |
| *** |-paul-| has joined #openstack-infra | 20:33 | |
| anteaya | Zara: fair enough | 20:34 |
| anteaya | mriedem: they appear to be separate patches to me, did I +A prematurely? | 20:35 |
| *** HeOS has joined #openstack-infra | 20:35 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Exclude gate-ironic-inspector-dsvm-ipa-nv from stable/kilo https://review.openstack.org/296772 | 20:35 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Exclude gate-tempest-dsvm-ironic-pxe_ipa from stable/kilo https://review.openstack.org/296773 | 20:35 |
| mriedem | anteaya: done | 20:35 |
| anteaya | you rebased 296773 ontop of 296772? | 20:36 |
| *** e0ne has quit IRC | 20:37 | |
| anteaya | oh sorry I see AJaeger's comment | 20:37 |
| *** Sukhdev has quit IRC | 20:38 | |
| *** tonytan_brb has joined #openstack-infra | 20:38 | |
| *** jsavak has quit IRC | 20:39 | |
| *** jsavak has joined #openstack-infra | 20:39 | |
| openstackgerrit | Travis Tripp proposed openstack-infra/project-config: Adding searchlight-ui repo and jobs https://review.openstack.org/293848 | 20:40 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add barbican client to project list for lbaas testing https://review.openstack.org/296777 | 20:40 |
| *** amuller has left #openstack-infra | 20:40 | |
| openstackgerrit | Travis Tripp proposed openstack-infra/project-config: Adding searchlight-ui repo and jobs https://review.openstack.org/293848 | 20:40 |
| *** arxcruz has quit IRC | 20:41 | |
| *** tonytan4ever has quit IRC | 20:42 | |
| *** sdague has quit IRC | 20:44 | |
| *** mylu_ has quit IRC | 20:44 | |
| fungi | mordred: not sure specifically what you wanted to discuss about secure apt and package index signing, but i figured we'd just create an openpgp key, stuff the secring.gpg file as an opaque blob into hiera and puppet it onto mirror-update.openstack.org for use by reprepro and whatever else needs it | 20:46 |
| openstackgerrit | Merged openstack-infra/project-config: Switch to mktemp for wheel-index.sh https://review.openstack.org/296716 | 20:46 |
| *** mylu has joined #openstack-infra | 20:46 | |
| mordred | fungi: yes! that is what I think too | 20:46 |
| fungi | mordred: then we can push the public key into sks and all sign it and push up our sigs | 20:47 |
| openstackgerrit | Merged openstack-infra/project-config: Use correct flavors for centos7 and fedora23 in rax https://review.openstack.org/296585 | 20:47 |
| mordred | fungi: I think what I more wanted to talk about was -- I think it might be time to do such a thing | 20:47 |
| fungi | mordred: seems like a fine time to me | 20:47 |
| mordred | fungi: turns out - our AFS mirrors are super useful even to not-us | 20:47 |
| fungi | okay | 20:47 |
| fungi | not sure how/to whom | 20:47 |
| *** BigWillie has quit IRC | 20:48 | |
| mordred | fungi: well, setting up a vhost that points at files in AFS on a machine with AFS client and alarge cache | 20:48 |
| mordred | fungi: is a _very_ quick way to get an excellent trusty mirror | 20:48 |
| fungi | oh, gotcha | 20:48 |
| mordred | it works so well, in fact, that I sort of want to blog about how to do it | 20:49 |
| fungi | possibly also a very quick way to put additional load on our afs fileservers, but i suppose "setting up an afs client" is a high enough bar we won't see a ton of that happening | 20:49 |
| mordred | but if I were to do that, I think having the repos not be signed goes from being a little weird to yuck | 20:49 |
| mordred | fungi: yah - and the client side caching makes the load also likely to not be terrible | 20:49 |
| *** angdraug has joined #openstack-infra | 20:50 | |
| fungi | also, i definitely don't want to be setting security expectations for those repos outside of our relatively innocuous use cases | 20:50 |
| *** kgiusti has left #openstack-infra | 20:50 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-openstackid: Order of the classes parameters is refactored https://review.openstack.org/295320 | 20:50 |
| clarkb | is there no way to have reprepro do a proper mirror and avoid signing at all? | 20:50 |
| fungi | but since we're signing it with clearly unofficial (for the distros we're mirroring) keys, that also seems realtively easily discoverable | 20:50 |
| clarkb | that seems like the best option? | 20:50 |
| anteaya | fungi: do you really want 295320 gumming up everything else with openstackid? | 20:51 |
| *** nadya has quit IRC | 20:51 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Use standard python3-jobs definition for oslo.messaging https://review.openstack.org/296374 | 20:51 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Fix voting for amqp1-centos oslo.messaging jobs https://review.openstack.org/296781 | 20:51 |
| anteaya | I did try to leave HeOS as message in channel to stop with the puppet refactoring | 20:51 |
| fungi | clarkb: guaranteeing your mirror is complete and that the indices you've mirrored match the packages you've mirrored is something we'd need to do separately if we're not generating our own indices | 20:51 |
| anteaya | but he didn't seem to get it yet | 20:51 |
| *** jpr has quit IRC | 20:52 | |
| clarkb | fungi: are the indexes signed too? | 20:52 |
| clarkb | or just the packages? if just the packages it seems like reprepro could leave it be and make a valid index | 20:52 |
| pleia2 | anteaya: yeah, can't say I'm thrilled with these changes | 20:52 |
| anteaya | pleia2: noone is | 20:52 |
| *** EricGonczer_ has joined #openstack-infra | 20:52 | |
| *** fawadkhaliq has quit IRC | 20:53 | |
| fungi | clarkb: the indices are all that get signed | 20:53 |
| anteaya | yet the author doesn't seem willing to cease | 20:53 |
| pleia2 | not sure what triggered them, and I don't think there was any discussion | 20:53 |
| HeOS | anteaya, I'm here. | 20:53 |
| clarkb | fungi: oh I thought it was the packages | 20:53 |
| fungi | clarkb: the indices in turn list checksums of the padckages | 20:53 |
| fungi | packages | 20:53 |
| anteaya | HeOS: okay so several folks have asked for the puppet refactor patches to cease | 20:53 |
| *** fawadkhaliq has joined #openstack-infra | 20:53 | |
| anteaya | HeOS: can you cease? | 20:53 |
| anteaya | HeOS: we have enough issues with openstackid as it is | 20:53 |
| clarkb | fungi: ok so you would need an inverse sort of tool from what reprepro gives | 20:54 |
| anteaya | and the storyboard folks need to review other patches, yet feel pressured to review yours so it doesn't get merged without review | 20:54 |
| clarkb | instead of making a valid index for some list of packages you need to ensure the list of packages is valid for the given index | 20:54 |
| HeOS | anteya, yes, I can, but I don't understand, how this affects openstackid. :( | 20:54 |
| clarkb | whihc is harder to do without breaking the world on every update | 20:54 |
| fungi | clarkb: basically, yes, you'd need to perform some manner of index validation (check sigs on indices and then check checksums of all files covered by them) before you vos release | 20:54 |
| openstackgerrit | Ed Leafe proposed openstack-infra/irc-meetings: Update the Nova Scheduler meeting https://review.openstack.org/296786 | 20:55 |
| anteaya | HeOS: are you not the author of this patch to puppet-openstackid? https://review.openstack.org/#/c/295320/ | 20:55 |
| *** shardy_ has quit IRC | 20:55 | |
| HeOS | anteaya, yes I'm the author. | 20:56 |
| anteaya | great | 20:56 |
| pleia2 | HeOS: we already have a large review queue of important changes, changing things that aren't broken, but have the potential to break things, just to align with style guidelines isn't currently a priority for us, and I think a massive set of patches like you proposed could have used some prior discussion | 20:56 |
| *** imcsk8 has quit IRC | 20:56 | |
| clarkb | fungi: and if you kept the old packages around for some period of time then in theory you could avoid the missing package problem | 20:56 |
| *** _ari_ has quit IRC | 20:56 | |
| pabelanger | I should not, that puppet-lint does have a check for display order of params. But it doesn't look to be updated to the latest syntax style guide. | 20:57 |
| pabelanger | http://puppet-lint.com/checks/parameter_order/ | 20:57 |
| pleia2 | pabelanger: yeah, but even for linting we tend to talk about things before making sweeping changes (we do ignore some lint things too) | 20:57 |
| *** tongli has quit IRC | 20:58 | |
| HeOS | pleia2, anteaya, I thinks, we can discuss it at openstak-infra ML, to make a decision how to deal with this. | 20:58 |
| anteaya | HeOS: if you like | 20:58 |
| anteaya | but folks are asking you to cease | 20:58 |
| HeOS | *think | 20:58 |
| pabelanger | pleia2: agreed. Just pointing out there maybe a time in the future where we either need to disable the check or contine the work HeOS is doing. | 20:58 |
| *** jamesmcarthur has quit IRC | 20:58 | |
| pleia2 | pabelanger: yep, fair enough :) | 20:58 |
| *** david-lyle has quit IRC | 20:59 | |
| AJaeger | pabelanger: for new changes, we could ask for the order to make it easier in the future... | 20:59 |
| pabelanger | I don't have a strong opinion on it, except yolanda already reviewed a bunch, so I followed suit with some +A | 20:59 |
| * AJaeger waves good bye | 20:59 | |
| pabelanger | AJaeger: I tend to default to that myself anyways. Just helps me find vars easier | 20:59 |
| pabelanger | AJaeger: later | 20:59 |
| anteaya | AJaeger: thanks, enjoy your evening | 21:00 |
| pleia2 | AJaeger: have a good one | 21:00 |
| *** jamesmcarthur has joined #openstack-infra | 21:00 | |
| crinkle | pabelanger: it's actually broken https://github.com/rodjek/puppet-lint/pull/429 | 21:00 |
| *** achanda has joined #openstack-infra | 21:01 | |
| pabelanger | crinkle: Ah | 21:01 |
| pabelanger | so if that lands, we'll feel some pain | 21:01 |
| *** aysyd has quit IRC | 21:01 | |
| fungi | or we'll (temporarily?) disable that check | 21:01 |
| crinkle | yes, but it's been sitting there for nearly a year so i don't expect the pain to come anytime soon | 21:01 |
| pabelanger | crinkle: agreed | 21:01 |
| pabelanger | fungi: and yes too | 21:01 |
| crinkle | also the alphabetical reordering is not part of that check or part of the style guide as far as i can tell | 21:02 |
| pabelanger | either way, having HeOS do work on our puppet modules is fine for me. As it was a stepping stone to get move involve with #infra :) So, that's why I was helping | 21:03 |
| *** rossella_s has quit IRC | 21:03 | |
| *** rossella_s has joined #openstack-infra | 21:03 | |
| fungi | i already find the => alignment insane. if all = statements are supposed to be aligned i'm likely to be largely infavor of just disabling the check indefinitely | 21:03 |
| HeOS | Yes, I want to make your manifests much better. :) For example, add docstrings there and so on. | 21:04 |
| *** fawadkhaliq has quit IRC | 21:04 | |
| *** fawadk has joined #openstack-infra | 21:04 | |
| *** oanson has quit IRC | 21:04 | |
| HeOS | I've made a decision to start my work from refactoring of ordering of the variables in the classes. | 21:05 |
| *** david-lyle has joined #openstack-infra | 21:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't build ironic ramdisk on postgres job https://review.openstack.org/296770 | 21:05 |
| fungi | i also don't see much point to alpha-ordering variables unless we've had a rash of people adding duplicate conflicting assignments or something. to find variables i use a keyword search in my editor | 21:05 |
| *** burgerk has quit IRC | 21:05 | |
| anteaya | HeOS: that is great that you want to help | 21:05 |
| *** jaypipes has quit IRC | 21:06 | |
| anteaya | I often find it a good place to start by asking "how can I help?" | 21:06 |
| fungi | i do see the listing of required parameters before optional parameters useful | 21:06 |
| pabelanger | I would be much happier if we gave HeOS a task then if replacement of the reordering of params, if s/he is up for it | 21:06 |
| anteaya | pabelanger: task away | 21:06 |
| *** _amrith_ is now known as amrith | 21:06 | |
| fungi | though we very rarely make use of required parameters and set defaults for just about everything | 21:06 |
| anteaya | I don't have anything I can monitor myself right now | 21:06 |
| *** sc68cal has quit IRC | 21:06 | |
| *** matrohon has joined #openstack-infra | 21:07 | |
| anteaya | but I am not in favour of any more hands in the pie in openstackid | 21:07 |
| *** dtardivel has quit IRC | 21:08 | |
| HeOS | Anyway, I'll write a letter on ML to discuss it and make a decision. :) It would be better, imho. | 21:09 |
| *** mylu has quit IRC | 21:09 | |
| *** nelsnelson has quit IRC | 21:10 | |
| *** nelsnelson has joined #openstack-infra | 21:10 | |
| jeblair | it may be better, but to do so means the lost opportunity to do something even better. | 21:10 |
| fungi | i'm agreeing with crinkle, i don't see anywhere at https://docs.puppetlabs.com/guides/style_guide.html that recommends alphabetical ordering of parameters | 21:11 |
| fungi | i'm also not seeing anything there that says = should be subject to columnar alignment, only => | 21:11 |
| *** thorst has quit IRC | 21:11 | |
| fungi | https://docs.puppetlabs.com/guides/style_guide.html#display-order-of-parameters "Should align hash rockets (=>) within blocks of attributes, remembering to arrange hashes for maximum readability first." | 21:12 |
| *** thorst has joined #openstack-infra | 21:12 | |
| *** mylu has joined #openstack-infra | 21:12 | |
| pabelanger | fungi: I am looking at https://docs.puppetlabs.com/guides/style_guide.html#display-order-of-parameters good example | 21:12 |
| pabelanger | which meets both of those things | 21:12 |
| pabelanger | even though they don't explicitly say it | 21:12 |
| fungi | pabelanger: the example may do those things, but that does not mean that the guide states they should be done | 21:12 |
| pabelanger | but I agree with jeblair, there might be better things to do. I'd just like us to say what that might be. I don't have anything off the top of my head for new contributors | 21:13 |
| pabelanger | fungi: true | 21:13 |
| fungi | it seems odd that they would have a preference to do columnar alignment for both = and => but only mention it (and as a should, not a must) for => | 21:13 |
| clarkb | kibana4, xenial testing if alpha or betas are out yet, both come to mind as theoretically straightforward that anyone can jump into | 21:14 |
| fungi | they do say required parameters before optional parameters is a must, and that makes sense | 21:14 |
| *** thorst_ has joined #openstack-infra | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** mylu has quit IRC | 21:15 | |
| *** EricGonczer_ has quit IRC | 21:16 | |
| *** tonytan_brb is now known as tonytan4ever | 21:16 | |
| matrohon | hi all, does anyone knows how to test a new job added in project-config, but still in review with a patch in the repo that the job is targeting? | 21:16 |
| *** EricGonczer_ has joined #openstack-infra | 21:16 | |
| matrohon | I've tried with "Depends-On" patch in the targeted repo, but it didn't work | 21:16 |
| *** thorst__ has joined #openstack-infra | 21:16 | |
| *** tpsilva has quit IRC | 21:17 | |
| matrohon | for instance : https://review.openstack.org/#/c/296515/2 | 21:17 |
| *** thorst has quit IRC | 21:17 | |
| anteaya | matrohon: the new job has to be merged and on our jenkins masters in order to run | 21:17 |
| anteaya | matrohon: if you have a new job you are creating, run it in the experiemental pipeline of your repo | 21:17 |
| anteaya | then trigger it with 'check experimental' as a comment on a patch to that repo | 21:18 |
| anteaya | that will make the experimental jobs run | 21:18 |
| matrohon | anteaya, I wanted to test my new experimental jobs before merging it! :) | 21:18 |
| anteaya | you can't | 21:18 |
| *** thorst_ has quit IRC | 21:18 | |
| anteaya | it has to be merged | 21:18 |
| matrohon | anteaya, ok thanks | 21:18 |
| anteaya | that is the only way jenkins can know about it to run it | 21:18 |
| anteaya | welcome | 21:18 |
| *** tmcpeak has joined #openstack-infra | 21:19 | |
| clarkb | anteaya: matrohon well you can run it locally | 21:19 |
| clarkb | without jenkins | 21:19 |
| tmcpeak | mordred: o/ | 21:19 |
| anteaya | matrohon: what clarkb said | 21:19 |
| clarkb | take the shell in https://review.openstack.org/#/c/258522/3/jenkins/jobs/networking-bgpvpn.yaml and run it against that change | 21:19 |
| openstackgerrit | Merged openstack-infra/project-config: Exclude gate-ironic-inspector-dsvm-ipa-nv from stable/kilo https://review.openstack.org/296772 | 21:20 |
| openstackgerrit | Merged openstack-infra/project-config: Exclude gate-tempest-dsvm-ironic-pxe_ipa from stable/kilo https://review.openstack.org/296773 | 21:20 |
| *** amrith is now known as _amrith_ | 21:20 | |
| anteaya | mriedem: ^^ | 21:20 |
| *** jimkenty has joined #openstack-infra | 21:20 | |
| *** thorst__ has quit IRC | 21:21 | |
| *** david-lyle has quit IRC | 21:21 | |
| jimkenty | Hi, does anybody know what version of Tempest is meant to work with stable/kilo OpenStack? | 21:21 |
| matrohon | clarkb, well it seems there are errors in our yaml! the gate might misinterpret our yaml. that 's what I want to tes | 21:21 |
| *** edmondsw has quit IRC | 21:22 | |
| matrohon | clarkb, I'll give it a try thanks | 21:22 |
| *** ddieterly has joined #openstack-infra | 21:22 | |
| *** edmondsw has joined #openstack-infra | 21:22 | |
| clarkb | matrohon: we do test that the yaml compies and show you the output | 21:22 |
| *** edmondsw has quit IRC | 21:22 | |
| clarkb | the underlying issue is jenkins jobs are basically privileged | 21:22 |
| clarkb | this means they need to be reviewed before running against a jenkins master | 21:23 |
| *** IlyaG has quit IRC | 21:23 | |
| matrohon | clarkb, I understand thansk for the explanation | 21:23 |
| fungi | jimkenty: we run tempest on stable/kilo changes all the time | 21:23 |
| *** david-lyle has joined #openstack-infra | 21:23 | |
| fungi | jimkenty: latest tempest should be fine for stable/kilo | 21:24 |
| fungi | jimkenty: but the #openstack-qa channel is where the tempest authors hang out and they can get into the weeds with you on tempest versioning and releases if you want | 21:24 |
| matrohon | clarkb, what do you mean by yaml output? where can I see it? | 21:24 |
| *** ociuhandu has joined #openstack-infra | 21:25 | |
| clarkb | matrohon: http://logs.openstack.org/22/258522/3/check/gate-project-config-compare-xml/0f11411/console.html | 21:26 |
| *** lakshmiS has quit IRC | 21:26 | |
| *** Sukhdev has joined #openstack-infra | 21:26 | |
| *** sdake has quit IRC | 21:27 | |
| mriedem | anteaya: thanks | 21:27 |
| anteaya | welcome | 21:28 |
| pabelanger | Woohoo | 21:28 |
| pabelanger | http://mirror.ord.rax.openstack.org/wheel/centos-7-x86_64/ | 21:28 |
| pabelanger | I think we did it | 21:28 |
| anteaya | congratulations | 21:28 |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job https://review.openstack.org/296808 | 21:29 |
| pabelanger | ianw: ^ | 21:29 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Added mobile detection for create user form https://review.openstack.org/296809 | 21:30 |
| ianw | cool, let's see if devstack can work now | 21:30 |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job https://review.openstack.org/296808 | 21:30 |
| matrohon | clarkb, thanks | 21:31 |
| *** regXboi has quit IRC | 21:31 | |
| *** _amrith_ is now known as amrith | 21:32 | |
| pabelanger | ianw: where does devstack setup wheel mirrors? | 21:33 |
| clarkb | pabelanger: ianw it doesnt we set it up in the nodepool ready script | 21:33 |
| ianw | it should be from the pip conf | 21:34 |
| clarkb | you could hack a patch to one off set it and test though | 21:34 |
| pabelanger | clarkb: ya, that's what I am looking at now | 21:34 |
| *** daemontool_ has quit IRC | 21:34 | |
| *** thorst has joined #openstack-infra | 21:34 | |
| jimkenty | fungi: thanks, i'll hop on that channel and see | 21:35 |
| *** baoli has quit IRC | 21:36 | |
| *** ddieterly is now known as ddieterly[away] | 21:36 | |
| *** tmcpeak has left #openstack-infra | 21:36 | |
| *** IlyaG has joined #openstack-infra | 21:37 | |
| *** daemontool_ has joined #openstack-infra | 21:38 | |
| pabelanger | clarkb: looking at https://git.openstack.org/cgit/openstack-infra/project-config/commit/nodepool/scripts/configure_mirror.sh?id=10602b13d9e152c571d96954e3bdf48e254b52df I'm not sure what else would need to be added for centos7 support | 21:38 |
| *** thorst has quit IRC | 21:38 | |
| clarkb | pabelanger: possibly nothing | 21:39 |
| pabelanger | okay | 21:39 |
| *** asettle has joined #openstack-infra | 21:40 | |
| jeblair | yeah, i think it should just start working as soon as there is content there | 21:40 |
| *** ddieterly[away] is now known as ddieterly | 21:40 | |
| pabelanger | exciting | 21:41 |
| openstackgerrit | Adrian Otto proposed openstack/requirements: Updated magnum client version to 2.0.0 https://review.openstack.org/296810 | 21:41 |
| *** |-paul-| has quit IRC | 21:42 | |
| *** amitgandhinz has quit IRC | 21:42 | |
| *** amrith is now known as _amrith_ | 21:42 | |
| fungi | jeblair: i have an odd situation in zuul's debug log. i uploaded 296708,2 to gerrit, zuul saw the event and processed it at 2016-03-23 20:23:42,188 but then it never enqueued it for testing, nor can i find that it said why it didn't | 21:43 |
| fungi | worried we're blind to something going on there | 21:43 |
| *** rhallisey has quit IRC | 21:44 | |
| clarkb | greghayn1: ianw SpamapS what are the chances of getting a new dib release soon? hoping to have the eth1 situation fixed | 21:44 |
| mordred | jeblair: do we need to do the first release of a wheel mirror by hand? | 21:44 |
| mordred | jeblair: or are they small enough that it doesn't matter | 21:44 |
| *** Nepoc has quit IRC | 21:45 | |
| *** roxanagh_ has joined #openstack-infra | 21:45 | |
| *** rocky_g has quit IRC | 21:45 | |
| greghayn1 | clarkb: I need to eat, but then I can release | 21:45 |
| clarkb | woot thanks | 21:45 |
| *** rocky_g has joined #openstack-infra | 21:45 | |
| openstackgerrit | Adrian Otto proposed openstack/requirements: Updated magnum client version to 2.0.0 https://review.openstack.org/296810 | 21:46 |
| *** achanda has quit IRC | 21:46 | |
| anteaya | fungi: you only caught that because you uploaded the patch? | 21:48 |
| fungi | anteaya: correct, i checked it to see what the check results were, noticed it never got any, and then went hunting in zuul logs | 21:48 |
| fungi | could be a limited issue, or could be representative of a larger situation | 21:48 |
| *** matrohon has quit IRC | 21:49 | |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: net-bgpvpn : fix access to rc files https://review.openstack.org/296812 | 21:49 |
| anteaya | so only one zuul log entry for that patch number? would there be any way to search the logs on incidents of a lone patch number? | 21:49 |
| *** roxanagh_ has quit IRC | 21:49 | |
| anteaya | because when zuul reported back it would have the patch number logged again, would it not? | 21:49 |
| fungi | not easily that i can think of. would require some multi-pass parsing/correlation of the zuul log i think | 21:50 |
| anteaya | ah okay | 21:50 |
| *** matrohon has joined #openstack-infra | 21:50 | |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: net-bgpvpn : fix access to rc files https://review.openstack.org/296812 | 21:51 |
| matrohon | clarkb, anteaya : ^ thanks for your help, I think we found our issue | 21:52 |
| *** zeih_ has quit IRC | 21:52 | |
| anteaya | yeah I'm looking at your patch | 21:53 |
| *** mriedem has quit IRC | 21:53 | |
| anteaya | I'm seeing a lot of use of /opt/stack/ hardcoded in project-config, but not yet convinced that is the best way to address what is happening | 21:53 |
| anteaya | could be a whole lot of cargo culting going on | 21:54 |
| clarkb | well $BASE is a devstack-gate thing so that part of the change is correct | 21:54 |
| anteaya | yeah his patch removes $BASE in favour of hard coding /opt/stack | 21:54 |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn-bagpipe jobs https://review.openstack.org/258522 | 21:55 |
| clarkb | yup because $BASE isnt a thing until you run devstack-gate | 21:55 |
| anteaya | ah okay thanks | 21:55 |
| *** doug-fish has quit IRC | 21:55 | |
| *** sdake has joined #openstack-infra | 21:55 | |
| *** ihrachys_ has quit IRC | 21:56 | |
| anteaya | clarkb: so you agree with the change? | 21:56 |
| clarkb | I havne't reviewed the whole thing so not sure | 21:57 |
| anteaya | okay | 21:57 |
| matrohon | clarkb, I'm just changing $BASE for /opt/stack in this change | 21:58 |
| *** IlyaG has quit IRC | 21:58 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:59 | |
| clarkb | mordred: oregon vs duke, is this going to be fun? | 21:59 |
| *** zeih has joined #openstack-infra | 21:59 | |
| *** hashar has quit IRC | 22:00 | |
| clarkb | matrohon: the change does a lot more, I haven't reviewed that change as a whole | 22:00 |
| anteaya | clarkb: enjoy the sports ball | 22:00 |
| *** jiggalator has joined #openstack-infra | 22:01 | |
| *** EricGonczer_ has quit IRC | 22:01 | |
| matrohon | clarkb, this change ? https://review.openstack.org/#/c/296812/ | 22:01 |
| mordred | clarkb: it's either going to be fun or terrible | 22:01 |
| clarkb | oh is that different than https://review.openstack.org/#/c/258522/4 ? | 22:02 |
| clarkb | I am clearly confused | 22:02 |
| matrohon | clarkb, 258522 depends on 296812 | 22:03 |
| matrohon | in 296812, I'm just fixing our mistake with the $BASE | 22:03 |
| jeblair | mordred: i think it's already happened, so... i guess it's not too big :) | 22:03 |
| matrohon | clarkb, in 258522, we're adding a new job | 22:03 |
| *** knikolla has quit IRC | 22:04 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Improve the tag-complete directive https://review.openstack.org/295452 | 22:04 |
| *** jamesmcarthur has quit IRC | 22:04 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add a resource abstraction for worklist filters https://review.openstack.org/295453 | 22:05 |
| *** HeOS has quit IRC | 22:06 | |
| jeblair | fungi: i don't think zuul did see the upload event for that change | 22:06 |
| jeblair | i think it only saw a comment-added event for it | 22:06 |
| fungi | jeblair: oh really? it mentioned noticing it as being in state new for that patchset | 22:06 |
| fungi | but maybe that was just for teh comment added event? | 22:07 |
| jeblair | yeah, i think so | 22:07 |
| fungi | so we may be missing gerrit event stream items from time to time | 22:07 |
| jeblair | fungi: perhaps moreso during network maintenance events | 22:09 |
| *** mylu has joined #openstack-infra | 22:09 | |
| *** baoli has joined #openstack-infra | 22:09 | |
| anteaya | I don't see in cacti that gerrit was experiencing a high load event an hour and a half ago | 22:10 |
| ianw | clarkb / greghayn1 : i really wish we took the time to get some consistency there. yet another case of elements walking over each other and inconsistent flags | 22:10 |
| clarkb | ianw: where? | 22:11 |
| ianw | in that eth* removal patch | 22:11 |
| *** greghayn1 is now known as greghaynes | 22:11 | |
| clarkb | I am not sure what consistency is missing? simple-init knows it will write those out at boot so it deletes them at image build time if they exist | 22:11 |
| greghaynes | ianw: sure, but it's not really a change in behavior, just a bug | 22:11 |
| *** matrohon has quit IRC | 22:12 | |
| clarkb | right it was always doing the deletes it just failed to do them sufficiently | 22:12 |
| clarkb | mnaser: any luck tracking that image upload error down? Can I remove that image file from the web server? | 22:14 |
| fungi | jeblair: i have a feeling gerrit may also fail to emit events some times. people keep reporting changes at random not getting announced to channels and a few times i've looked and find no evidence of them in the gerritbot logs | 22:15 |
| ianw | because minimal creates them, then centos&fedora have flags to not create them, which are opposite, and ubuntu doesn't have a flag at all, and simple init shouldn't actually assume it will take over those interfaces -- it might be vaild that i only want simple-init to do one interface. i.e. a mess | 22:16 |
| anteaya | fungi: do you think that is new behaviour with 2.11? | 22:16 |
| *** achanda has joined #openstack-infra | 22:16 | |
| clarkb | ianw: the first half of that problem seems like a distro issue? eg if ubuntu and debian deboostrap them then thats what we get? | 22:17 |
| ianw | greghaynes: let's make this the last release without release notes? -> https://review.openstack.org/#/c/292193/ | 22:17 |
| fungi | anteaya: no clue really... insufficient data | 22:17 |
| anteaya | fungi: fair enough, I don't have a sense either | 22:17 |
| clarkb | ianw: for the other I don't know when you would want to do that but its certainly a possibility | 22:17 |
| *** jcoufal has quit IRC | 22:18 | |
| clarkb | unfortunately that possibility currently flies in the face of why glean exists at all | 22:18 |
| *** zeih has quit IRC | 22:18 | |
| clarkb | which is make eth0 and eth1 work in rackspace | 22:18 |
| *** baoli has quit IRC | 22:18 | |
| tonyb | greghaynes: Not sure what I'm asking really but .... Was the run to create new images successful last night? Is it work me trying my locale bug again? | 22:20 |
| fungi | tonyb: what was the job that was failing and what was the fix proposed to gerrit for it? | 22:21 |
| greghaynes | tonyb: I was wrong - we need to release dib for the change to go in images | 22:21 |
| fungi | question answered | 22:21 |
| fungi | no point in me checking ages of image uploads for now i guess | 22:22 |
| tonyb | greghaynes: Ahh okay. I'll try locally then | 22:22 |
| anteaya | tonyb: hey I had said I would help with irc meeting patches when the skipped meeting functionality was in | 22:22 |
| anteaya | tonyb: should I be waiting or no? | 22:22 |
| tonyb | fungi: this is the bug I'm tryign to fix: https://bugs.launchpad.net/openstack-gate/+bug/1560808 | 22:22 |
| openstack | Launchpad bug 1560808 in OpenStack-Gate "stable/liberty: gate-requirements-integration-dsvm fails with due to UnicodeDecodeError in httpretty" [Undecided,Confirmed] | 22:22 |
| openstackgerrit | Merged openstack-infra/openstackid: Added mobile detection for create user form https://review.openstack.org/296809 | 22:23 |
| tonyb | fungi: I *think* there was a DIB review that would fix it .... (listed in the bug) | 22:23 |
| *** dimtruck is now known as zz_dimtruck | 22:23 | |
| tonyb | anteaya: I think all the bits are in place for that release to happen. just waiting for Mitaka to release befoer altering libraries (which is probably overly pedantic on my part) | 22:24 |
| anteaya | tonyb: never overly pedantic about releases | 22:24 |
| anteaya | I support your judgement here | 22:25 |
| anteaya | thanks for having the bits in place :) | 22:25 |
| tonyb | anteaya: my aim is for you (and others) to be able to use it to skip meetings during the summit | 22:25 |
| tonyb | anteaya: only 6months too late | 22:25 |
| anteaya | oh I hope so | 22:25 |
| anteaya | never too late | 22:25 |
| anteaya | I'm going to offer a patch with this summit, next summit and christmas and new years | 22:26 |
| *** antonym has quit IRC | 22:26 | |
| anteaya | to ensure folks have lots of advance warning | 22:26 |
| tonyb | anteaya: cool. | 22:26 |
| anteaya | anytime is the right time for me | 22:26 |
| anteaya | so thank you | 22:26 |
| *** antonym has joined #openstack-infra | 22:26 | |
| anteaya | very grateful | 22:26 |
| greghaynes | ianw: about release notes - are you saying I should merge that and then release or wait until after rlease? | 22:26 |
| tonyb | anteaya: If you like tiou can propose it now understanding that it will fail until I do "the release" | 22:27 |
| ianw | greghaynes: i'm not super concerned now or later, but i'd like to see us keeping them soon | 22:27 |
| *** jsavak has quit IRC | 22:27 | |
| greghaynes | ianw: ok, I will release now since theres a few folks but I'll check out the reno stuff | 22:28 |
| greghaynes | er, a few folks waiting on it | 22:28 |
| pabelanger | infra-root: just confirmed with ianw, but centos7 wheel mirror appears to be working for devstack! Results will be posted to 295062! | 22:28 |
| *** sdake_ has joined #openstack-infra | 22:29 | |
| clarkb | pabelanger: the devstack log should record how much time it spent running pip, you could compare the new runs to the old runs to see how that changed | 22:29 |
| anteaya | tonyb: okay I'll put something up that will be wrong and then I can get the format straightened out in review, thank you | 22:29 |
| greghaynes | clarkb: tonyb ok, tag pushed | 22:30 |
| tonyb | greghaynes: Thanks. | 22:30 |
| anteaya | tonyb: have you any documentation on format? | 22:30 |
| *** sdake has quit IRC | 22:30 | |
| pabelanger | clarkb: I cheated and SSh'd into the jenkins slave. Looks like pip_install - 212 secs | 22:30 |
| ianw | clarkb: yeah, it's ~850 down to ~200 | 22:30 |
| clarkb | nice | 22:30 |
| pabelanger | ya | 22:30 |
| *** chem has quit IRC | 22:31 | |
| ianw | and we can still pull some more out of that i think with a few extra bits that devstack is still building | 22:31 |
| tonyb | anteaya: in the review, let me find a rendered vresion .... | 22:31 |
| *** daemontool_ has quit IRC | 22:31 | |
| tonyb | anteaya: https://github.com/openstack-infra/yaml2ical#example-2 | 22:32 |
| *** tonytan4ever has quit IRC | 22:32 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-guide https://review.openstack.org/296821 | 22:32 |
| anteaya | tonyb: thank you | 22:33 |
| tonyb | anteaya: you're welcome. | 22:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-guide https://review.openstack.org/296821 | 22:35 |
| *** jimkenty has quit IRC | 22:35 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Allow filters to be set on automatic worklists https://review.openstack.org/295454 | 22:36 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Enable tlsproxy by default https://review.openstack.org/296771 | 22:37 |
| pvaneck | jeblair: Hey, could you revisit https://review.openstack.org/#/c/281737 when you get a chance? | 22:38 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on revoke UI (Admin) https://review.openstack.org/296824 | 22:38 |
| *** rocky_g has quit IRC | 22:39 | |
| *** knikolla has joined #openstack-infra | 22:40 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/irc-meetings: Skip some meeting dates: third-party-meeting https://review.openstack.org/296825 | 22:41 |
| *** smarcet has quit IRC | 22:41 | |
| anteaya | tonyb: I have New Year's as a value in the yaml file, do you think I need to put that in double quotes? | 22:42 |
| tonyb | anteaya: I *think* it's okay. | 22:42 |
| anteaya | let's find out | 22:42 |
| tonyb | anteaya: just building a testenv | 22:43 |
| anteaya | cool | 22:43 |
| *** tristanC has quit IRC | 22:43 | |
| anteaya | can you show me how? | 22:43 |
| *** amitgandhinz has joined #openstack-infra | 22:43 | |
| *** tristanC has joined #openstack-infra | 22:44 | |
| *** Qiming has joined #openstack-infra | 22:44 | |
| *** pradk has quit IRC | 22:45 | |
| pabelanger | #success wheel-mirror-centos-7-amd64 is now online and serving wheels | 22:46 |
| tonyb | anteaya: sure just distilling the steps | 22:47 |
| anteaya | tonyb: thanks | 22:47 |
| *** rguillebert has quit IRC | 22:47 | |
| *** dingyichen has joined #openstack-infra | 22:48 | |
| *** Jeffrey4l has quit IRC | 22:49 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on revoke UI (Admin) https://review.openstack.org/296824 | 22:51 |
| tonyb | anteaya: git review -d 296825 # Optional | 22:51 |
| tonyb | anteaya: sed -i~ -e 's,yaml2ical,git+https://git.openstack.org/openstack-infra/yaml2ical@master#egg=yaml2ical,' test-requirements.txt | 22:51 |
| tonyb | to use the unreleased yaml2ical | 22:51 |
| tonyb | anteaya: tox -re ical,check | 22:51 |
| tonyb | run the tests (making sure you rebuild the env with the new yaml2ical | 22:52 |
| anteaya | cool thanks | 22:52 |
| tonyb | anteaya: FWIW the generated ics looks good to me. | 22:53 |
| *** fawadk has quit IRC | 22:53 | |
| anteaya | yay | 22:53 |
| *** fawadkhaliq has joined #openstack-infra | 22:53 | |
| tonyb | anteaya: I probably need to added them to the eavesdrop.o.o as well as the ics | 22:54 |
| anteaya | yes | 22:54 |
| *** yamahata has joined #openstack-infra | 22:54 | |
| anteaya | I point everyone at the eavesdrop index page | 22:54 |
| *** fawadkhaliq has quit IRC | 22:55 | |
| *** fawadkhaliq has joined #openstack-infra | 22:56 | |
| tonyb | anteaya: okay I'll work out how to do that befoer I release yaml2ical. | 22:56 |
| tonyb | anteaya: but that wont be this week. | 22:56 |
| anteaya | yup, thanks | 22:56 |
| anteaya | where do you see the generated ics in tox? | 22:56 |
| *** amitgandhinz has quit IRC | 22:57 | |
| anteaya | tonyb: ^ | 22:57 |
| tonyb | anteaya: output/calendars/third-party-meeting.ics | 22:57 |
| anteaya | thanks | 22:57 |
| tonyb | anteaya: np | 22:57 |
| anteaya | I was looking in .tox | 22:57 |
| *** Qiming has quit IRC | 22:58 | |
| anteaya | tonyb: the VALUE=DATE-TIME:20160425T080000Z | 22:59 |
| *** sdake_ is now known as sdake | 22:59 | |
| *** IlyaG has joined #openstack-infra | 22:59 | |
| anteaya | tonyb: I just put the date, do you think I need to add the time? or does it understand the 24 hour period from the default timestamp | 22:59 |
| tonyb | greghaynes: I'm getting duplicate elements: http://paste.openstack.org/show/491651/ DIB==1.13.0 | 23:00 |
| tonyb | anteaya: it gets it from the time element in the yaml | 23:00 |
| *** roxanagh_ has joined #openstack-infra | 23:00 | |
| anteaya | tonyb: great, thank you | 23:01 |
| tonyb | anteaya: so what you've done is great (adding the time would make it fail) | 23:01 |
| *** mriedem has joined #openstack-infra | 23:01 | |
| anteaya | ah, I won't do that then | 23:01 |
| *** ddieterly has quit IRC | 23:02 | |
| *** angdraug has quit IRC | 23:02 | |
| *** imcsk8 has joined #openstack-infra | 23:03 | |
| *** rbrndt_ has quit IRC | 23:03 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add release notes job for Kolla repo https://review.openstack.org/296129 | 23:04 |
| *** IlyaG has quit IRC | 23:04 | |
| *** peterlisak has quit IRC | 23:04 | |
| *** roxanagh_ has quit IRC | 23:05 | |
| *** alaski has quit IRC | 23:05 | |
| *** greghaynes has quit IRC | 23:05 | |
| *** verdurin has quit IRC | 23:05 | |
| *** darrenc_ has joined #openstack-infra | 23:05 | |
| *** flwang2 has quit IRC | 23:06 | |
| *** darrenc has quit IRC | 23:06 | |
| *** mattoliverau has quit IRC | 23:06 | |
| *** mylu has quit IRC | 23:06 | |
| *** darrenc_ is now known as darrenc | 23:07 | |
| *** matt6434 has joined #openstack-infra | 23:07 | |
| *** fawadkhaliq has quit IRC | 23:07 | |
| *** mylu has joined #openstack-infra | 23:07 | |
| *** fawadkhaliq has joined #openstack-infra | 23:08 | |
| *** weshay has quit IRC | 23:08 | |
| *** cdent has quit IRC | 23:09 | |
| openstackgerrit | Merged openstack-infra/system-config: Adding meetbot and statusbot support for openstack-product https://review.openstack.org/275364 | 23:09 |
| pabelanger | clarkb: sorry, I forgot to thank you in my ML post. I'll be sure to get you a beer in austin | 23:09 |
| pabelanger | for the help with the jenkins server | 23:09 |
| clarkb | pabelanger: heh np | 23:09 |
| clarkb | pabelanger: I will come to you for help on the afs front soon :) | 23:09 |
| *** alaski has joined #openstack-infra | 23:11 | |
| *** peterlisak has joined #openstack-infra | 23:11 | |
| *** zeih has joined #openstack-infra | 23:12 | |
| *** BigWillie has joined #openstack-infra | 23:12 | |
| *** gordc has quit IRC | 23:14 | |
| *** BigWillie has quit IRC | 23:16 | |
| *** erlon has quit IRC | 23:17 | |
| *** zeih has quit IRC | 23:18 | |
| *** greghaynes has joined #openstack-infra | 23:19 | |
| *** verdurin has joined #openstack-infra | 23:20 | |
| *** okay22many has quit IRC | 23:20 | |
| *** openstack has joined #openstack-infra | 23:23 | |
| ianw | pabelanger: https://github.com/pypa/pip/pull/3586 should fix "--log" for pip, so we can make the output a bit more manageable | 23:25 |
| pabelanger | ianw: looking | 23:26 |
| ianw | i think we should probably build each wheel separately in the job, and stick something easily greppable in the output | 23:26 |
| ianw | and i'll setup a monitoring job to warn us when we've got broken builds | 23:26 |
| *** sridhar_ram has quit IRC | 23:27 | |
| pabelanger | Ya, I wouldn't know which way is better to be honest. I'm sure others will pipe up however | 23:28 |
| ianw | #infra having an opinion? never :) | 23:29 |
| clarkb | is the issue that some wheels fail? | 23:29 |
| *** tonytan4ever has joined #openstack-infra | 23:29 | |
| clarkb | so for package in packages: pip wheel $package is better logging wise than pip wheel $packages? | 23:30 |
| pabelanger | clarkb: once we land https://review.openstack.org/#/c/296097/ that will help I think | 23:31 |
| pabelanger | but we are seeing 1 failure | 23:31 |
| pabelanger | Failed to build PyECLib | 23:31 |
| clarkb | oh that hasn't landed | 23:31 |
| pabelanger | on stable branches | 23:31 |
| clarkb | wow | 23:31 |
| clarkb | pabelanger: you can self approve it has the +2s it needs | 23:32 |
| pabelanger | clarkb: ok | 23:32 |
| clarkb | not sure why yolanda didn't approve | 23:32 |
| clarkb | I am impressed that without that we have working wheel mirror :) | 23:32 |
| pabelanger | indeed | 23:32 |
| pabelanger | I think in the morning, I'll go through the log and make sure we are not missing any libraries | 23:33 |
| pabelanger | but for now, I call it a night | 23:33 |
| * pabelanger & | 23:33 | |
| fungi | fwfw, if wheel building fails i _believe_ the requirements jobs will also break | 23:35 |
| fungi | but then again, we're probably only running those on ubuntu | 23:35 |
| clarkb | fwiw we still have 31 image uploads for the day running/queueud | 23:36 |
| anteaya | :( | 23:36 |
| *** ybathia has joined #openstack-infra | 23:36 | |
| clarkb | I am going to kick off a new ubuntu-trusty image build so that the later uploads upload an image with working eth1 | 23:37 |
| anteaya | yay working eth1 | 23:38 |
| clarkb | then once we manage to get that image uploaded everywhere we should be safe to reenable multinode on rax and switch the puppet jobs off devstack-trstuy then we can stop uploading devstack-trusty | 23:38 |
| openstackgerrit | Merged openstack-infra/system-config: Fix centos7 issue with wheel_mirror.pp https://review.openstack.org/296097 | 23:38 |
| *** fawadkhaliq has quit IRC | 23:39 | |
| anteaya | yay | 23:40 |
| clarkb | 2016-03-23 23:37:53,200 INFO nodepool.image.build.ubuntu-trusty: E: Couldn't download dists/trusty/main/binary-amd64/Packages | 23:40 |
| clarkb | lets try that again | 23:40 |
| anteaya | :( | 23:40 |
| odyssey4me | hmm, we have a job which doesn't seem to be getting picked up by zuul - https://review.openstack.org/296490 - we've tried rebasing and rechecking - can anyone take a peek into it? | 23:40 |
| clarkb | thats persistent woo | 23:41 |
| odyssey4me | lol, ok nvm the author abandoned it | 23:41 |
| clarkb | odyssey4me: usually that happens when you have a circular dependency tree | 23:42 |
| jhesketh | infra-root: tonyb has asked if we can rebuild ubuntu-trusty to get the latest disk-imagebuilder in to check if it fixes bug 1560808. Any objections to me firing off a rebuild? | 23:42 |
| openstack | bug 1560808 in OpenStack-Gate "stable/liberty: gate-requirements-integration-dsvm fails with due to UnicodeDecodeError in httpretty" [Undecided,Confirmed] https://launchpad.net/bugs/1560808 | 23:42 |
| clarkb | jhesketh: see scrollback :) | 23:43 |
| anteaya | odyssey4me: the parent change of that stack can't merge: https://review.openstack.org/#/c/272743/4 | 23:43 |
| clarkb | jhesketh: the last ~4 minutes | 23:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Error tracking and log save for wheel build https://review.openstack.org/296841 | 23:43 |
| clarkb | jhesketh: tl;dr is yes trying to but dib is failing because it can't get ubuntu packages | 23:43 |
| jhesketh | clarkb: heh, this is what I get for not keeping up with irc | 23:43 |
| odyssey4me | anteaya oh wow, lol - sorry about that and thanks :) | 23:43 |
| anteaya | odyssey4me: no worries, nice to see you | 23:43 |
| anteaya | and you're welcome | 23:43 |
| pabelanger | okay, not full gone | 23:44 |
| pabelanger | clarkb: https://review.openstack.org/#/c/294339/ should help with upload threads | 23:44 |
| clarkb | jhesketh: my initial reading is this is a network issue? | 23:44 |
| *** angdraug has joined #openstack-infra | 23:45 | |
| clarkb | jhesketh: I don't think this is related to the new dib release because its just running debootstrap | 23:45 |
| *** harlowja_at_home has quit IRC | 23:45 | |
| jhesketh | I wouldn't imagine those errors would be related, but it's hard to tell | 23:46 |
| *** harlowja_at_home has joined #openstack-infra | 23:46 | |
| jhesketh | clarkb: are you rebuilding again? | 23:46 |
| clarkb | jhesketh: I have tried 3 times now, hav enot tried a fourth time | 23:46 |
| clarkb | let me queue it again | 23:46 |
| clarkb | nerp still fialing | 23:47 |
| clarkb | http://archive.ubuntu.com/ubuntu/dists/trusty/main/binary-amd64/ | 23:49 |
| *** fawadkhaliq has joined #openstack-infra | 23:49 | |
| clarkb | in there are Packages.gz and Packages.bz2 the error is E: Couldn't download dists/trusty/main/binary-amd64/Packages | 23:49 |
| jhesketh | clarkb: hmm... do the sources look correct? | 23:49 |
| clarkb | ya if I grab the gz and the bz2 both looks fine at a quick galnce | 23:50 |
| *** jiggalator has quit IRC | 23:51 | |
| clarkb | does anyone know if it needs a literal 'Packages' file? | 23:52 |
| anteaya | I do not know | 23:53 |
| *** amitgandhinz has joined #openstack-infra | 23:53 | |
| *** eil397 has quit IRC | 23:53 | |
| *** sridhar_ram has joined #openstack-infra | 23:53 | |
| lifeless | Nakato: stevenk: either of you have b/w right now? I found a bug pbr bug. | 23:54 |
| lifeless | https://bugs.launchpad.net/pbr/+bug/1561254 | 23:54 |
| openstack | Launchpad bug 1561254 in PBR "sem-ver processing fails when a tag is not in canonical form" [High,Triaged] | 23:54 |
| jhesketh | clarkb: ah, yeah, I don't know either | 23:54 |
| StevenK | lifeless: I'm about to duck out to shop, WRT bandwidth to investigate, yes I do | 23:54 |
| tonyb | clarkb: IIRC it only uses Packages as a fallback if the otehr can't be found. | 23:54 |
| lifeless | StevenK: cool thanks | 23:55 |
| clarkb | http://paste.openstack.org/show/491658/ is the full relevant piece of log | 23:55 |
| clarkb | there is an aweful lot of whitespace but that should be fine | 23:55 |
| clarkb | git diff in the debootstrap element of dib shows only that a cache var was removed | 23:55 |
| *** Sukhdev has quit IRC | 23:57 | |
| tonyb | clarkb: so I ran it (the debootstrap) and it works | 23:58 |
| Nakato | lifeless: Yea | 23:58 |
| tonyb | clarkb: http://paste.openstack.org/show/491659/ | 23:59 |
| clarkb | tonyb: oh wow I see the problem this is amazing | 23:59 |
| tonyb | clarkb: good amazing? | 23:59 |
| clarkb | /dev/mapper/main-opt 504G 504G 1.5M 100% /opt | 23:59 |
| tonyb | clarkb: Oh | 23:59 |
| clarkb | I just cleaned /opt out a few days ago and we had like 200gb | 23:59 |
| *** Sukhdev has joined #openstack-infra | 23:59 | |
| clarkb | so not sure what exploded | 23:59 |
| *** ajmiller has quit IRC | 23:59 | |
| *** sridhar_ram has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!