21:00:20 #startmeeting Solum Team Meeting 21:00:20 Meeting started Tue Feb 24 21:00:20 2015 UTC and is due to finish in 60 minutes. The chair is devkulkarni. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:25 The meeting name has been set to 'solum_team_meeting' 21:00:33 #topic Roll Call 21:00:40 Devdatta Kulkarni 21:00:42 Melissa Kam 21:00:45 Akshay Chhajed 21:00:57 Ravi Sankar Penta 21:01:02 james li 21:01:03 Ed Cranford 21:01:03 Gilbert Pilz 21:01:13 hi everyone 21:01:29 #link https://wiki.openstack.org/wiki/Meetings/Solum#Weekly_Solum_Team_Meeting here is our agenda for today 21:01:43 please take a moment to go over it 21:01:58 while we wait to see if anybody else joins us 21:02:46 philip cheong 21:02:52 hi phiche 21:02:59 hello :) 21:03:07 #topic Announcements 21:03:14 just made it back to the hotel in time 21:03:21 nice :) 21:03:39 I would like to continue from last week where we had started on recent feature updates 21:03:56 last week muralia walked us through the custom language pack support 21:04:13 today I want to hand over to james_li for some of the work that he landed recently 21:04:19 (james_li): Robustness improvements 21:04:33 sure. One of my recent works is fix db concurrency issue: https://review.openstack.org/#/c/149062/ 21:04:55 we were seeing random functional tests failures on solum gate 21:05:18 sometimes an assembly just cannot be deleted 21:05:49 the root cause was the db update code in solum was not atomic 21:06:25 how so? could you give an example? 21:06:34 so that patch basically put the db update options into a single transaction/session 21:06:52 devkulkarni: yea 21:06:55 yeah 21:07:26 see we want to update the status of an assembly 21:08:00 what we did is that first do a query to db and load the row to a sqlalchemy object 21:08:10 then update the object, 21:08:17 then writes back to db 21:08:30 but not in a single session. 21:08:48 if a delete operation comes in between we got the problem 21:09:52 thats why we saw *random* failures on solum gate 21:10:02 ah !! okay.. so the delete will load the object as well and then if the first operation updates after the delete is complete the row will be back in the table. 21:10:03 james_li: we can have lost update problem but why the delete op not successful? 21:10:32 ravips: yes 21:10:56 my another patch solves the problem by retrying 21:11:07 https://review.openstack.org/#/c/154296/ 21:11:11 ravips: I think the way we check in the functional test whether a delete was successful 21:11:21 is to do a GET for it 21:11:34 james_li: devkulkarni: yeah, understood 21:11:40 ravips: cool 21:12:02 james_li: nice work on that patch. 21:12:03 ok 21:12:15 I remember seeing lot of gate failures before that patch landed 21:12:21 yes 21:12:29 about the retry logic 21:12:37 could you talk more about it.. 21:12:41 sure 21:13:12 if two db transactions wants to lock the same row 21:13:32 what db engine does is to let one go through 21:13:45 and throws an exception for another one 21:14:31 so the retry mechanism is as simple as catch the error (i.e. DBDeadlock and StaleDataError) and do a retry of the operation 21:14:47 ok, got it 21:14:52 nova and glance do the similar thing 21:15:04 I just borrow some code from them :) 21:15:10 :) 21:15:29 makes sense to add retry logic 21:15:40 thanks james_li for both the patches 21:15:48 np 21:15:59 moving on to the next feature update 21:16:04 this one also goes to james_li 21:16:09 (james_li): Custom port support 21:16:23 my ongoing work is to have a configurable port mapping 21:16:42 https://review.openstack.org/#/c/158482/ 21:17:28 solum needs to know the ports that apps listen on because solum needs to publish ports from app's docker container to VM 21:17:53 like: docker run -p 80:80 -p 443:443 21:18:20 The patch provides a easy way for users to config the ports. 21:18:32 so these will be the ports on the DU? 21:18:40 and it is under review now 21:18:59 devkulkarni: yes 21:19:10 cool 21:19:13 so ports are a sub-node of 'artifact' 21:19:23 gpilz: yes 21:19:35 makes sense 21:19:50 why multiple ports james_li? 21:20:09 some app listens on multiple ports, like solum itself 21:20:17 SSL vs non-SSL is one example 21:20:30 ok, makes sense 21:20:42 nice stuff james_li 21:20:51 james_li: app show will also display the ports for the app? 21:21:02 eventually 21:21:08 ravips: thats the client side change 21:21:13 not yet done 21:21:23 ok 21:21:31 james_li: what will be the du_url? 21:21:40 which port will that have? 21:22:07 I think we just return the IP? 21:22:13 as user knows the ports 21:22:36 right, thats what we currently return (so port is 80) 21:22:46 yes 21:22:51 your custom port is the port from the container 21:22:55 ok 21:23:22 ravips, datsun180b: makes sense to send info about all the relevant ports in app show 21:23:35 devkulkarni: i think user might be running any service, can be http/mysql or something on the given port and its app specific 21:24:07 agreed. and in that case we will have to modify the du_url to include the right port. right? 21:24:38 devkulkarni: du_url is for triggering builds? 21:25:02 ravips: no, du_url is actually the app url. the url of the deployed application 21:25:23 for triggering builds, we have the notion of a trigger_url 21:25:23 devkulkarni: how do we know the endpoint for the user app? 21:26:25 so currently we get the IP from heat stack-create, and that IP is essentially the user app endpoint. 21:26:33 devkulkarni: may plan file can have optional param to specify the endpoint and platform can return /: 21:26:53 +1 21:27:25 so the endpoint in this sense is sort of a context root? 21:27:29 why make the client do the work of figuring out IP + port + resource path 21:27:34 ? 21:28:00 devkulkarni: yes 21:28:20 yeah, specifying the required ports and the app url in planfile/or command line args makes sense 21:28:34 but for that we need backend support 21:28:36 whereby 21:28:57 our heat templates need to change to accommodate dns resource 21:29:13 but, conceptually I don't see a problem in the proposal 21:29:54 any more thoughts on this feature? if not, I can continue with updates on deployments 21:30:19 (devkulkarni): Supporting different backends for deployment 21:30:23 devkulkarni: we need port and endpoint/context-root association defined in the plan file otherwise when user specifies multiple ports, we don't know which port belongs to which endpoint 21:30:50 ravips: yes, agreed. 21:31:52 james_li: what do you think? we could restrict to one port for now 21:32:06 since we don't yet have the dns resource in our HOT 21:32:24 devkulkarni: sure 21:33:20 so essentially, the custom port will be the port on the container, which is always mapped to same port on the VM. Then, the application url is just the IP address that we get back from heat and the port 21:33:51 ravips, james_li: thoughts? 21:34:13 +1 21:34:26 devkulkarni: for the first pass, instead of showing app url we can display ip and exposed ports to the user 21:35:02 one problem I see is with applications that have more than one artifact 21:35:13 ravips: but then how would the user get the application url? how do they know which port to use? 21:35:20 but maybe we should leave that discussion for another time 21:35:57 devkulkarni: my assumption here is user has defined the ports, given the ip user should be able to deduce the url 21:36:02 gpilz: yes, we need few other things in place before we can tackle multiple artifact use case 21:37:31 ravips: when you say, user has defined the ports, that means user has somehow specified that a particular port, say 5000, is where the application will be available on, vs, another port, say 4444, is where the JMX agent will be replying to 21:37:49 but for that we need to introduce the required mechanisms in the plan file and/or on the CLI 21:38:48 devkulkarni: james_li current work support specifying ports in the plan file right? 21:39:05 james_li: how much effort do you see in introducing some sort of indicator in the plan file 21:39:29 ravips: yes 21:39:43 devkulkarni: don't understand 21:40:50 can we make the simplifying assumption that every artifact will result in one enpoint that may have more than one port? 21:40:51 james_li: what I meant is, to support what ravips is saying, for each port we would need to indicate whether it is application url port vs. port for some other aspects of the application 21:41:07 so planfile says "88, 451" 21:41:37 this results in URIs http://somenode.org:88/foo/buzzle 21:41:48 http://somenode.org:451/foo/buzzle 21:41:54 sorry "https"? 21:42:25 how did you know https? 21:42:27 nevermind 21:42:28 devkulkarni: i think the ports defined in plan file are just for app itself 21:43:02 gpilz: sure. and I am saying, we need some way to distinguish that port 88 is http vs. port 541 is for https. 21:43:09 right, exactly 21:43:18 it seems like there are two pieces of this problem 21:43:44 one is creating the necessary mappings so you an actually connect to the application in whatever way you are supposed to connect to it 21:43:55 and the other is advertising those endpoints 21:44:11 seem like james' patch addresses the first problem but not the second 21:44:19 ? 21:44:39 ravips: I am working with the view point that solum needs to tell the app deployer the application url (http://somenode.org:88), and that it should not be the application developer's responsibility to construct this url 21:44:39 gpilz: yes 21:45:07 ravips - i agree, but i don't think we should lay that work on this patch 21:45:43 deffo later 21:45:59 I am also of the opinion that it could be done in a later patch 21:46:08 gpilz: james_li: yes, this can be done later 21:46:20 i think current work by james_li is the first step 21:46:30 ravips: cool. yep, this is the first step in that direction 21:46:58 wow! its almost end of the meeting time.. 21:47:16 devkulkarni: someone can create a blueprint to support further use cases 21:47:31 ravips: makes sense. 21:47:49 I am inclining towards postponing further updates to next meeting and jumping to Blueprint/task review 21:48:01 there is one particular item I want to get to today 21:48:09 since we have ravips and akshayc with us 21:48:28 what do you all think about jumping to task review? 21:48:44 +1 21:48:47 +1 21:48:48 proceed 21:48:54 ok then.. 21:49:09 #topic Blueprint/Task Review 21:49:12 (devkulkarni): Converting bash scripts to Python in build-app, unittest-app, build-lp 21:49:25 Here is the latest on this 21:49:49 we found a way to get around on the dependency issue 21:49:56 In our testing, we are now realizing that the fact that our core engine not being in python has started to hurt us 21:50:26 and so we need these to be converted to python for better testing, logging, etc. 21:50:38 there are few things already in place: 21:50:44 1) set of patches by ravips 21:51:06 2) akshayc's willingness to take over that work from ravips 21:51:22 3) some of the blockers, like ability to use dockery-py 21:51:54 and there is one more thing to that list 21:52:10 4) from rackers, james_li is also interested in helping out with that work 21:52:40 about the overall approach — 21:52:50 I see this can be tackled in two ways.. 21:53:05 One approach is what ravips has put forth. this is a top-down approach 21:53:22 where we start with changing the interface between shell.py and the bash scripts 21:53:35 Another approach is to do a inside-out approach. 21:53:56 where we don't change the interface first, but start with changing small little things inside those scripts 21:54:04 an example of this is: 21:54:20 https://review.openstack.org/#/c/157594/ 21:54:39 I feel that the inside-out approach will give us incremental way to move forward 21:55:04 thoughts? 21:55:52 what do you mean by inside out? start using python in bash scripts first? 21:56:08 yes 21:56:17 without changing the interface. 21:56:30 the main advantage here is it is in small incremental steps 21:56:47 I don't know if the final shape of these changes would line up with ravips's proposed changes 21:56:55 ok…. that might work…. 21:57:04 and I agree that ravips's work is where we want to eventually reach 21:57:05 devkulkarni: good compromise, minimal hindrance to the rest of the development 21:57:08 ya…. the last step will be big 21:57:14 cool 21:57:34 james_li: thx for signing up for this task 21:57:35 so that was the main point I wanted to bring up. 21:57:57 we have 3 minutes remaining 21:58:05 let me just start the open discussion topic 21:58:10 #topic open discussion 21:59:46 ok then.. 21:59:53 thanks everyone for joining today 22:00:01 we will continue with feature updates in the next meeting 22:00:04 #endmeeting