| *** zhiwei has joined #openstack-chef | 01:03 | |
| *** zhiwei has quit IRC | 03:00 | |
| *** zhiwei has joined #openstack-chef | 03:00 | |
| *** openstackstatus has quit IRC | 03:01 | |
| openstackgerrit | Samuel Cassiba proposed openstack/cookbook-openstack-common: Adds distro-specific changes to make CI work https://review.openstack.org/291921 | 04:45 |
|---|---|---|
| openstackgerrit | Samuel Cassiba proposed openstack/cookbook-openstack-common: Adds distro-specific changes to make CI work https://review.openstack.org/291921 | 05:25 |
| sc` | markvan: the glance issue isn't the user :( | 05:37 |
| sc` | even as admin it throws 403s | 05:37 |
| sc` | however, centos works fully after a workaround for removing the pip installed requests module | 05:38 |
| sc` | @core we need to start landing some patches to move on now that ci is working, mostly | 05:41 |
| os-chef-bot | @j^2 @markvan @mattray @wenchma @jklare @cmluciano @zhiwei @sc` we need to start landing some patches to move on now that ci is working, mostly | 05:41 |
| *** nkrinner has joined #openstack-chef | 05:49 | |
| *** openstackstatus has joined #openstack-chef | 08:28 | |
| *** ChanServ sets mode: +v openstackstatus | 08:28 | |
| -openstackstatus- NOTICE: jobs depending on npm are now working again | 08:31 | |
| *** ChrissKO_ has joined #openstack-chef | 09:51 | |
| frickler | who is channel-admin and can fix the topic? statusbot somehow messed that up | 10:03 |
| *** zhiwei has quit IRC | 10:44 | |
| *** ChrissKO_ has quit IRC | 10:57 | |
| *** jhesketh has joined #openstack-chef | 11:17 | |
| *** ChanServ changes topic to "OpenStack & Chef | chef.io/openstack | bit.ly/os-chef-wiki | bit.ly/os-gerrit-inbox | bit.ly/os-chef-repo | bit.ly/os-chef-meetings-logs | bit.ly/os-chef-meeting-agendas | bit.ly/stackalytics-chef" | 11:17 | |
| *** jhesketh has quit IRC | 12:07 | |
| *** jhesketh has joined #openstack-chef | 12:07 | |
| *** ChrissKO_ has joined #openstack-chef | 12:30 | |
| *** zhiwei has joined #openstack-chef | 12:53 | |
| *** zhiwei has quit IRC | 12:57 | |
| *** os-chef-bot has quit IRC | 13:04 | |
| *** os-chef-bot has joined #openstack-chef | 13:05 | |
| markvan | sc`: yeah, I can't repoduce the glance issue locally yet, but I still think we should not be using the service creds for image uploads. I'll update that commit msg | 13:58 |
| openstackgerrit | Mark Vanderwiel proposed openstack/cookbook-openstack-image: Use admin user for glance uploads https://review.openstack.org/301892 | 13:59 |
| *** nkrinner has quit IRC | 14:12 | |
| *** Ryjedo has joined #openstack-chef | 14:25 | |
| openstackgerrit | Mark Vanderwiel proposed openstack/openstack-chef-repo: Use default cirros image defined in cookbook https://review.openstack.org/302810 | 14:30 |
| sc` | markvan: yeah. no problem. we should be using admin to upload to glance, but the error almost makes me think policy.json | 14:35 |
| markvan | sc`: After digging a bit more, I think we have a timing issue in the image recipe, I see in the u14 logs a log called glance-api.log, that is the default | 14:50 |
| markvan | sc`: but when we lay down our cookbook api.conf, the log file is api.log. So on u14 I think the packaging is starting the service during apt install. | 14:51 |
| sc` | aha. i did see that log file, but i hadn't connected the two yet | 14:51 |
| sc` | so we just call our log file glance-api.log and it should be fine | 14:52 |
| markvan | That means that when we lay down our conf, the service is already started, and then the recipe sets up the service and "starts" it, but it's already started (with the bad default conf) | 14:52 |
| markvan | in liberty we had this the other way around, the conf files would notify the service, now we have the service subscribes to the conf files | 14:53 |
| sc` | right. i see what you're saying. it's pre-coffee for me | 14:53 |
| markvan | so that only will work, if the service is NOT started by default. | 14:53 |
| markvan | but I don't understand why it works locally for me, all I can come up with right now is timing, that it's fast enough to lay down the conf file before the service it spun up | 14:55 |
| sc` | interesting. only the glance startup bits are in glance-api.log. api.log only has the db migrate | 14:55 |
| sc` | and the times are within the same minute | 14:55 |
| markvan | yup, compare those times to when the conf is laid down. | 14:56 |
| sc` | if it's getting a bad conf file first, that could explain the cryptic missing db connection messages | 14:56 |
| markvan | yup, that's my theory | 14:57 |
| *** jmb__ has quit IRC | 14:59 | |
| sc` | yeah. that's a solid theory. i just debunked my policy.json idea | 14:59 |
| sc` | markvan: what about adding a :delayed in the subscribes? | 15:02 |
| markvan | but since the api.log is created, it seems like the api service has been restarted to pick that up, but you'll notice there is no registry.log, so maybe that service is still "old" | 15:03 |
| sc` | in liberty, we were using :immediately, :delayed, etc. with notifies | 15:03 |
| markvan | yeah, seems like we can't trust what state the services will be in after packaging install, need to make sure our conf's are taking effect | 15:07 |
| markvan | sc`: Ah, found the proof (I think), search the main log for glance-registry, you will see on the ubuntu side, it's "up-to-date" because it was started by packing install, but on c7, it's enabled and started, so picks up the our conf. | 15:10 |
| markvan | sc`: so, I think we need to revisit how we are using subscribes as this may effect other cookbooks as well. | 15:12 |
| markvan | ah, and look at the u14 log, you will see that the glance-registry does get restarted, but at the end, and AFTER the image upload was tried and failed. | 15:14 |
| markvan | sc`: so, since we are using the glance services within the cookbook them selves, we probably need to restart those with :immediate instead of the default of :delay | 15:15 |
| markvan | I think that's how we handle keystone since we use that service as well | 15:15 |
| markvan | yup, it's there: https://github.com/openstack/cookbook-openstack-identity/blob/master/recipes/server-apache.rb#L311 | 15:16 |
| markvan | I'll push up a patch for making the glance services restart immediately | 15:17 |
| sc` | i can tack on yet another Depends-On and see if we can get working integration | 15:18 |
| markvan | :) | 15:18 |
| sc` | centos works, with a workaround for the base image. i was chatting with one of the infra folks a bit yesterday | 15:18 |
| sc` | puppet uses the same workaround, so we're not the unique ones | 15:19 |
| sc` | i recall in liberty, that notifies were tightly controlled, but that didn't make it over to mitaka yet | 15:22 |
| openstackgerrit | Mark Vanderwiel proposed openstack/cookbook-openstack-image: Restart glance immediately https://review.openstack.org/302866 | 15:24 |
| markvan | sc`: ^^ | 15:24 |
| markvan | sc`: since this is only required by image_upload, the other approach would be to put a notify in that recipe such that this is only done when actually needed, thoughts? | 15:26 |
| markvan | humm, but that notify has no knowledge to if the restart is really needed or not, so would force it each time...not what we want. | 15:27 |
| markvan | so, we could make the immediate flag conditional within the api/registry recipes based upon the image_upload T/F flag, a bit much? | 15:28 |
| * markvan mtg time for me... | 15:28 | |
| *** zhiwei has joined #openstack-chef | 15:42 | |
| *** jmb__ has joined #openstack-chef | 15:42 | |
| sc` | markvan: for mitaka? probably shouldn't invest a whole heap more of dev time | 15:43 |
| sc` | may be something to spend some cycles on for newton and backport | 15:44 |
| *** zhiwei has quit IRC | 15:47 | |
| openstackgerrit | Samuel Cassiba proposed openstack/cookbook-openstack-common: Adds distro-specific changes to make CI work https://review.openstack.org/291921 | 15:55 |
| markvan | sc`: yup, I agree, for mitaka this immed image patch should do the trick for us. | 16:28 |
| sc` | centos int just completed successfully, again | 16:39 |
| sc` | ubuntu still enqueued | 16:39 |
| *** zhiwei has joined #openstack-chef | 16:57 | |
| *** zhiwei has quit IRC | 17:02 | |
| sc` | rdo packages are soon upon us. testing is available, and the release will go out when it's validated in rdo's ci | 17:10 |
| sc` | since i already have an open common change, i'm going to flip to that | 17:10 |
| sc` | already see one thing -1 worthy. didn't dry the release name in the url | 17:16 |
| sc` | markvan: ubuntu seems to be working. it's going through the post install smoke tests and no issues are being encountered | 17:23 |
| sc` | that's pre my changes | 17:24 |
| sc` | one with my changes is still enqueued | 17:24 |
| sc` | but that says it's getting past the 403 by restarting glance immediately | 17:24 |
| sc` | no reason to think it won't work post my changes | 17:25 |
| markvan | sc`: that's good news...it's time to get this next chunk done | 17:27 |
| sc` | yup | 17:27 |
| sc` | i'll of course hold my champagne and #success until 291921 has finished | 17:27 |
| *** jmickle has joined #openstack-chef | 17:41 | |
| sc` | i see one problem with centos. there's some /etc/apt stuff that gets created | 17:44 |
| sc` | i'll see if it does it locally. if it's only in ci, i don't care as much | 17:45 |
| sc` | markvan: looks like ubuntu converge works, too. 291921 is on track to be all green | 18:48 |
| *** jmb__ has quit IRC | 18:57 | |
| sc` | and there it is. https://review.openstack.org/#/c/291921/ is all green | 19:09 |
| sc` | now we need to land some patches | 19:10 |
| *** zhiwei has joined #openstack-chef | 19:29 | |
| *** zhiwei has quit IRC | 19:34 | |
| markvan | sc`: woohoo! | 19:41 |
| *** The_8th_Dwarf has joined #openstack-chef | 19:50 | |
| *** jmb__ has joined #openstack-chef | 20:39 | |
| *** shakamunyi has joined #openstack-chef | 21:17 | |
| *** markvan has quit IRC | 21:24 | |
| *** shakamunyi has quit IRC | 21:24 | |
| *** markvan has joined #openstack-chef | 21:27 | |
| *** ChrissKO_ has quit IRC | 21:28 | |
| *** shakamunyi has joined #openstack-chef | 22:18 | |
| *** shakamunyi has quit IRC | 22:24 | |
| *** zhiwei has joined #openstack-chef | 22:41 | |
| *** zhiwei has quit IRC | 22:47 | |
| *** Daviey has joined #openstack-chef | 22:49 | |
| *** ChrissKO_ has joined #openstack-chef | 23:01 | |
| *** ChrissKO_ has quit IRC | 23:06 | |
| *** Ryjedo has quit IRC | 23:22 | |
| *** Ryjedo has joined #openstack-chef | 23:24 | |
| *** Ryjedo has quit IRC | 23:43 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!