Tuesday, 2015-12-22

docaedoAlso the dead-link checker is holding for https://review.openstack.org/#/c/260230/ to merge. jenkins-slaves need requestsexceptions (I failed to check that manifest for the slave...)00:22
docaedoIn all my tests so far though, it works great, finds the bad, and updates the yaml.00:22
docaedoof course for that test to be of any real use we need to settle the new shed color on https://review.openstack.org/#/c/257663/ ;)00:23
docaedo(and I'm still happy with using attributes field for this and a few other things under the assumption we'll NOT be using this method after glare, which might even be in before next summit!)00:24
*** kzaitsev_mb has quit IRC01:34
*** kzaitsev_mb has joined #openstack-app-catalog04:55
*** kzaitsev_mb has quit IRC06:34
*** kzaitsev_mb has joined #openstack-app-catalog06:37
*** kzaitsev_mb has quit IRC06:43
*** kzaitsev_mb has joined #openstack-app-catalog06:53
*** kzaitsev_mb has quit IRC07:03
*** kzaitsev_mb has joined #openstack-app-catalog07:04
*** kzaitsev_mb has quit IRC07:09
*** kzaitsev_mb has joined #openstack-app-catalog09:10
*** kzaitsev_mb has quit IRC14:53
*** kzaitsev_mb has joined #openstack-app-catalog17:00
kfox1111morning. looking17:17
kfox1111+2'ed the kloudbuster image. I'll let you workflow when the image is ready in storage.17:18
kfox1111What do you think of the code change I suggested in the yaml2json.py?17:20
kfox1111do you have a review up for the link checker itself yet?17:21
*** kzaitsev_mb has quit IRC17:23
openstackgerritMerged openstack/app-catalog: Update KloudBuster Image to v5  https://review.openstack.org/25965917:23
docaedoDead link checker job is merged as an experimental job: https://review.openstack.org/260115 but it needs this one to merge: https://review.openstack.org/26023017:24
docaedoI used requestsexceptions but that didn't happen to be on the slaves (and I assumed it was as infra said "oh you just need to use requestexceptions here!")17:25
docaedobut I think that'll merge today or so, then test it with adding a comment "check experimental" to any review17:25
docaedoin theory, should result in proposal-bot suggesting a new yaml, but I'll stay on it17:26
kfox1111https://review.openstack.org/#/c/260115 is not it.17:27
docaedore: the code change in yaml2json, I think it still confused me, maybe it was suggested in the wrong place? If the idea was just to shorten the script by two lines, I'm kind of .. eh :)  But maybe there's a different/better reason17:27
kfox1111just easier to read to me. just does the filtering in one spot.17:27
docaedooops, this one https://review.openstack.org/259232 is the dead link checker but that particular patch was missing a critical line that got added in 26011517:28
kfox1111instead of adding 8 lines, it just adds one.17:29
docaedoyou sure? Still need to expand the heat environment section I think17:30
kfox1111wow.... there's a lot of code in check_app_catalog_yaml.17:30
kfox1111I'm just counting green lines.17:31
kfox1111if you look at the left hand side,17:31
kfox1111all you would have to do is add the one line I suggested at the point I suggested to do the same thing I think.17:31
kfox1111y = [a for in in y] makes a new list with every member of y in it.17:32
kfox1111y = [a for a in y] I mean.17:32
kfox1111y = [a for a in y if ...] adds a into the new list then, only if it matches the conditional.17:32
kfox1111so it does all the same filtering using a new list the other code does. just a lot less code.17:33
kfox1111is all that yaml code just to get it not to munge the yaml file?17:34
docaedoyes, all that is needed to keep the yaml in the same order17:35
kfox1111woudl have been way easier with just dumping a second yaml and merging it.17:35
kfox1111would have been 2 lines in the yaml2json script instead of one.17:35
kfox1111deactivated = yaml.load(open('linkchecker.yaml'))17:36
kfox1111y = [a for a if a not in deactivated]17:36
kfox1111hmm... well...17:37
kfox1111y = [a for a if a['name'] not in deactivated]17:37
docaedomaybe, but that won't help me when I start checking hashes17:37
kfox1111why not?17:38
docaedoactually, maybe it wouldn't hurt .. I guess I'm still opposed to multiple yamls for this, rather than keeping everything in one17:39
kfox1111just makes for a lot more code, trying to ensure the code doesn't munge the human written yaml.17:40
kfox1111having 2 lets you have the machine generated one where no one cares about ordering stuff, and one for humans that do.17:40
docaedoeasier for humans to look in one yaml and see their asset has been marked inactive though17:41
* kfox1111 shurgs.17:41
kfox1111I figured thats what the email thing woudl be fore.17:41
kfox1111getting a notification by email's a lot easier for a human then watching a gerrit review board.17:42
kfox1111meeting next. bbiab.17:42
docaedowe're nowhere near being able to mail people when their asset has a dead link, and all in that would be way more work than something that works right now17:43
docaedolike literally, all of what has been written, works, and achieves the goal of automatically updating the catalog to remove dead links17:44
kfox1111its great step in the right direction to get us the ability to remove dead liks.17:57
kfox1111but thats about all we can do with it. it will be hard most of the time for us to update links. :/17:57
*** openstackgerrit has quit IRC18:02
*** openstackgerrit has joined #openstack-app-catalog18:03
docaedowhen you say "hard to update links", are you talking about keeping hashes up to date for things like fedora beta, debian, etc?18:40
docaedoalso keep in mind, I expect all of this to need to be re-though/re-worked when glare is implemented, but I'm very much looking forward to ditching the yaml entirely :)18:45
kfox1111keeping a hash up to date can be automated with the link checker. if it finds a hash that doesn't match, it can post the update as part of the gerrit review. odds are if the hash matches more then once a new file, thats the correct hash. I'm worried more about the bad link case, where they move the images around. :/19:17
kfox1111then it might not be clear the right thing for us to do. it would take the origional submitter or someone else knowlegable on that image to post a new update, which would mean we would have to contact them somehow to let them know it broke.19:18
docaedoI don't follow though, when you say "move the image around"? who's doing that and expecting the catalog to keep it up to date for them19:19
kfox1111thats the problem. if the person who maintains the app catalog entries for an image is a different person then maintains the site hosting the image, things can get "cleaned up" / moved around on the site. left hand vs right hand kind of thing.19:20
kfox1111leaves stale entries in the app catalog.19:21
docaedobut if the link dies because someone moved it, then the link is dead, and inactivated, right?19:21
kfox1111but we don't nessisarilyh have the knowlege to fix it and reenable it. while as the maintainer does. hense the need to email them. :/19:23
docaedoyes - I agree that's an issue, but I don't see that being an issue in the next 2-3 months do you?19:23
kfox1111no idea. will see I guess once the link checker's in place.19:23
docaedoAs great as it would be, I don't envision a sudden massive influx of content into the catalog. I like what we're doing, but it still sits in a very odd place, where we want to deliver what you would get from a PaaS application catalog, but without a PaaS19:25
kfox1111right. :/19:26
kfox1111I think the biggest problem though is lack of pluing support in any of the distro's (except debian) yet.19:26
kfox1111though I don't think anyone uses debian except for testing at home. :/19:26
kfox1111if a user cant easily find any of it, then its unlikely for a developer to bother to post anything to the catalog. :/19:27
kfox1111we've gotten the plugin into rdo-delorean, but I don't think its in the liberty repo like I was hoping.19:27
kfox1111haven't had time to go figure out how to make that happen. :/19:28
kfox1111we also need to make sure it gets into packstack, rdomanager, into ubuntu, and to fuel.19:28
openstackgerritChristopher Aedo proposed openstack/app-catalog: Skip assets with active==false attribute  https://review.openstack.org/25766319:28
kfox1111then we'll have some visibility. :/19:28
docaedo^^ is the script with shorter lines, finally grok'd what you were getting at, and agree :)19:29
docaedoer, not shorter lines - fewer lines. you know what I mean19:29
docaedoagree that the horizon plugin would change things for sure, and have kept "improving adoption" in the back of my mind...19:30
kfox1111you pulled too much code... got rid of the env stuff...19:31
kfox1111let me take a quick stab at it...19:31
docaedoShould also see some good movement on general catalog adoption when IBM includes it in bluemix. Don't know the scale of the audience, but it could be significant, and would be the biggest pickup of the catalog yet.  Hoping still for first quarter next year19:31
docaedoI think the env stuff still worked though, at least when I looked at the output, it was there and formatted as expected19:31
kfox1111yeah, that would be nice.19:31
docaedounless the heat environment formatting needs something even more special?19:31
kfox1111no, I think it will break. it won't be formatted as a json string.19:32
docaedoservice": {"environment": {"parameters": {"db_port": 1000}}, "type": "heat", "format": "HOT"}19:33
kfox1111yeah. should be {"environment": "{\"parameters\": {\"db_port\": 1000}}", "type": "heat", "format": "HOT"}19:34
docaedoah got it .. well that will help for all those people deploying the hello world template :)19:37
kfox1111it was easier to do it that way. the javascript in the plugin just needs a string, and can pass it along without converting json to a string.19:37
kfox1111and the user can just put yaml into the yaml file as expected and it will convert properly.19:38
docaedosure but you know - that comes at the expense of like .. THREE EXTRA LINES in that yaml script!19:38
kfox1111yeah, but way more then 3 lines to fix on the js side. ;)19:39
kfox1111needed like a whole nother external js file. :/19:39
docaedoI'm going to be AFK for a while, but will check back in on this later, I'm still holding out hope that dead link checker is implemented before 2016 :D19:39
kfox1111still think skipping attributes is a good idea too.19:40
kfox1111why bother to implement a schema if we ignore it?19:40
openstackgerritKevin Fox proposed openstack/app-catalog: Skip assets with active==false attribute  https://review.openstack.org/25766319:54
kfox1111it checks for things like active: "false" and rejects them.19:55
kfox1111hmm.... you know, the other interesting way to do this would be with a tag...19:57
kfox1111we could then allow users to still search on and see inactive things, but default them to hidden.19:57
kfox1111if we bump asset version to 2 when adding the tags: ['inactive'], existing horizon plugins will just not show the asset at all. giving the correct behavior. and then in an updated horizon plugin, we could provide a checkbox to show inactive assets that would know asset version v2.20:01
kfox1111hmm... actually, the code right now looks like it still shows the asset, but marks it as disabled, so you can still find it.20:36
kfox1111damn... found a bug in the horizon plugin..20:54
kfox1111I could have sworn I tested schema versions...20:55
kfox1111I think dep handling broke it. :/21:01
docaedoScore one for free form attributes field :D haha21:13
kfox1111no, free form wouldn't have the desired affect either.21:53
kfox1111basically I was thinking of this: http://pastebin.com/3JXRg4KM21:55
kfox1111if it wasn't for the horizon plugin bug, that would have worked too.21:56
docaedoOR we just exclude inactive assets from the JSON and call it a year21:57
kfox1111we only have one horizon plugin version that's affected by it. since we have the python api in place now, we could fix it like in the patch, and filter out all version>2 stuff in the api if horizon plugin version 1.0.0 is found.21:57
kfox1111I'm partial to tagging it so that others that want to go look for broken things can.21:58
kfox1111they can easily setup cron jobs for example to check if their assets are marked inactive in the server.21:58
kfox1111we can then put reasons things are disabled too in the asset. like, checksum mismatch, or broken link.21:59
docaedothey can do the same with a cron job hitting the repo, or by subscribing to app-catalog updates in which case they'll see the email21:59
docaedoI'm not proposing the checksum updater download every asset and do the hash22:00
kfox1111I really don't think a git repo should be an api endpoint.22:00
kfox1111email might work, but an api endpoint would work better with nagios for example.22:00
kfox1111then you can mail the oncall person for example.22:00
kfox1111ah. so its just checking for 404's? any plan to do checksum checking too?22:01
docaedohow many tb of bandwidth per day should the catalog be allocated? :P22:01
docaedofor checking, it's checking basically for a 200 response, or a 301/302 (in which case it will follow that looking for 200(22:02
kfox1111good question.22:02
kfox1111depends I guess  on the cloud providers...22:03
docaedoalso downloading and comparing the hash against expected is a waste of time - it will only reflect that one download that the catalog did22:03
kfox1111with amazon, bandwidth in is free, so download checking would be free.22:03
docaedowould have no reflection on what the end user gets - it's on the user to download and compare the hash22:03
kfox1111yeah, but it woudl detect cases where they updated image a with image b at the same location, but didn't update the catalog with the new sum.22:04
docaedo(which is something glance should probably do - take a remote URL and a hash, and compare the hash before importing)22:04
kfox1111the user thinkgs its a bad link, and its true in a wy. they don't know if its new or tampered with.22:04
kfox1111agreed. glance artifacts shoudl have checksumming built in I think.22:04
docaedothats walking too close to guaranteeing what's in the catalog (vs. making a best effort to keep an index of things up to date)22:04
* kfox1111 shrugs.22:05
kfox1111daily checking links, weekly checking checksums... seems best effortish to me.22:05
kfox1111its validating what we believe the metadata in our repo is reasonably current.22:06
docaedoit still solves little because the user HAS to check when they get the asset, in this case it just makes the catalog be a middleman that makes a claim about a hash - the user still has to compare the hash22:06
docaedoI can see that as a mechanism for confirming the metadata is accurate22:06
docaedobut then it becomes very costly, unless the app catalog is hosting the binary assets (in which case they would have been hash-verified upon entry anyway)22:07
docaedowould be awesome if there was an http header for hash :)22:07
kfox1111right. if we're paying for bandwidth, its too costly probably. if its free, I'm less sure.22:08
kfox1111yeah. that would be awesome. :)22:08
docaedosomeone is paying for the bandwidth, and when it becomes excessive that someone stops giving hosting away for free22:09
kfox1111yeah, but take amazon for example. I figured out the reason I think they give away free bandwidth.22:10
docaedowhat free bandwidth does amazon give away?22:10
docaedoyou mean like the first gb free stuff?22:10
kfox1111netflix essentially pays for the majority of the bandwidth out of amazon. they have to run wires to cover the cloud->internet direction.22:10
kfox1111that means they have a ton of bandwidth going from the internet->cloud that's unused.22:11
kfox1111so they give it away. it doesn't cost them much.22:11
kfox1111if you upload stuff into aws, they don't charge you per gig.22:11
kfox1111downoad, they do.22:11
kfox1111it all depends on how much unused capacity they have, in which directions.22:12
kfox1111netflix is almost 37% of all internet traffic.22:13
docaedoah sure but they always make it free or cheap to bring in, because your stuff is captive then, the easier and cheaper it is to move in the better (as long as you have to pay through the nose to move out)22:13
kfox1111that means the corisponding upload to the aws cloud pipe must be huge and relatively idle. :)22:13
kfox1111right. but with talking to them, they don't seem to care, if you pull data in, process it down to something small, and output that.22:14
kfox1111or just chew on it and never spit anything back out.22:14
kfox1111they have so much, they really just don't seem to care when we've talked to them.22:14
kfox1111so putting a checkummer in aws for example, woudl be just the cost of the vm doing the summing, not the bandwidth.22:15
kfox1111I kind of wonder which other cloud providers run into that situation.22:15
kfox1111most cloud hosts are hosting websites, which is bandwidth cloud->internet, so a very similar issue.22:15
kfox1111internet->cloud's much less common, so may be cheeper/free.22:16
docaedoI'm going to be mostly away for a while now :) Will check back in on occasion - hope you have great holidays!22:19
kfox1111yup. have a great one! :)22:21
openstackgerritChristopher Aedo proposed openstack/app-catalog: Add dead-link test script to tools  https://review.openstack.org/26019823:15

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!