Thursday, 2019-06-06

*** roman_g has quit IRC00:01
*** michael-beaver has quit IRC00:08
*** aaronsheffield has quit IRC00:11
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Adds unit tests for Spyglass CLI  https://review.opendev.org/66314000:45
*** kskels has quit IRC01:49
*** AlexNoskov has quit IRC02:11
*** cheng1 has quit IRC04:11
*** aojea has joined #airshipit06:23
*** pgaxatte has joined #airshipit06:29
*** jamesgu__ has quit IRC07:14
*** roman_g has joined #airshipit07:52
openstackgerritKudaka Poorna Rajesh proposed airship/promenade master: Add liveness/readiness probes to calico-etcd  https://review.opendev.org/65986308:11
*** happyhemant has joined #airshipit08:19
*** happyhemant has quit IRC10:38
*** bastianstola has joined #airshipit12:24
*** alexanderhughes has joined #airshipit12:43
*** aaronsheffield has joined #airshipit12:51
openstackgerritMerged airship/spyglass-plugin-xls master: Include examples in package  https://review.opendev.org/66333712:55
openstackgerritIan Pittwood proposed airship/spyglass master: Schema validation  https://review.opendev.org/65917212:56
openstackgerritIan Pittwood proposed airship/spyglass master: Manifest undefined data validation  https://review.opendev.org/65568312:57
*** bastianstola has quit IRC13:00
*** bastianstola has joined #airshipit13:03
openstackgerritMatt McEuen proposed airship/promenade master: Change CoreDNS from Deployment to DaemonSet  https://review.opendev.org/66244813:11
openstackgerritMatt McEuen proposed airship/promenade master: Change CoreDNS from Deployment to DaemonSet  https://review.opendev.org/66244813:15
*** howell has joined #airshipit13:19
*** kranthikirang has joined #airshipit13:26
*** mloza has joined #airshipit13:29
openstackgerritMerged airship/pegleg master: Update tests dir to use yaml.safe_load()  https://review.opendev.org/66276913:29
*** vladyslav has quit IRC13:30
*** kguttikonda has joined #airshipit13:30
*** kranthikirang has quit IRC13:34
*** aojea has quit IRC13:48
*** AOjeaGarcia has joined #airshipit13:48
*** kskels has joined #airshipit13:56
*** nishantkr has joined #airshipit14:00
*** jamesgu_ has joined #airshipit14:02
*** MattCoachCarter has joined #airshipit14:53
*** bastianstola has quit IRC15:07
openstackgerritJiri Suchomel proposed airship/shipyard master: Get password for shipyard user for special variable if it exists  https://review.opendev.org/66369815:17
mattmceuenIn case anyone didn't notice -- the airship-in-a-bottle project is renamed back:  https://opendev.org/airship/airship-in-a-bottle15:21
mattmceuen(thanks openstack infra team!)15:21
mattmceuenThe "in-a-bottle" project redirects15:21
kguttikonda@team, how can I communicate to drydock-api in UCP after having X-Auth-Token?15:31
kguttikondawhat else I need to add in headers in order to get nodes ?15:31
roman_ghttps://review.opendev.org/#/c/661004/ - review, please15:31
kguttikondadocumentation says: Secured endpoints require Keystone authentication and proper role assignment for authorization15:32
kguttikondabut no example; can someone help me here?15:32
kskelshere is example from library that is used in our CICD - https://github.com/att-comdev/cicd/blob/master/vars/shipyard3.groovy15:33
kskelsit's not super clear cut code but it does uses API of shipyard directly15:34
kskelsit's generally a standard openstack flow - get keystone token and then pass it to shipyard API15:34
roman_g            'Content-Type': 'application/json',15:34
roman_g            'X-IDENTITY-STATUS': 'Confirmed',15:34
roman_g            'X-USER-NAME': 'Test',15:34
roman_g            'X-ROLES': 'admin'15:34
roman_gI'm not sure, it's just my guess.15:35
roman_gI've taken this headers from drydock tests15:35
kguttikondaso; we don't have to pass  X-Auth-Token in headers?15:36
kguttikondaX-USER-NAME is drydock or Test? Sorry, I am confused15:36
evgenylkguttikonda: You have to pass X-Auth-Token, because it uses a standard keystone middleware as far as I know.15:38
evgenylkguttikonda: But first you will need to request a token from ucp Keystone.15:38
kguttikondaI did pass that and still getting {15:38
kguttikonda    "message": "Unauthenticated",15:38
kguttikonda    "type": "error",15:38
kguttikonda    "retry": false15:38
kguttikonda}15:38
evgenylkguttikonda: Are you sure that the token you requested is correct? It's the way cicd script you referenced above does it https://github.com/att-comdev/cicd/blob/master/vars/shipyard3.groovy#L6515:40
evgenylI'm not sure what is `X-Context-Marker` though.15:40
*** pgaxatte has quit IRC15:41
kguttikondaperhaps X-Context-Marker required for shipyard but I am trying to drydock-api here;15:41
kguttikondaDoes it need something else?15:41
evgenylkguttikonda: Oops, sorry, I thought you needed Shipyard api, and not drydock..15:42
kguttikondaperfect; it works now; Apparently I am using wrong x-Auth-Token15:43
kguttikondaSorry for that; my bad15:43
kguttikondaOne of my compute node fails at MAAS deployment; Apparently network configured correctly; and it fails to download bootactions/units15:44
kguttikondaany idea where to check? Other computes are deployed and joined to k8s15:44
kguttikondaI am not sure what is wrong here; hence trying to explore drydock-api15:44
kguttikondaapi/v1.0/bootactions/nodes/compute-5/units15:48
evgenylkguttikonda: Can you tell us a bit more about the failure, where did it fail and can you share the error message?15:49
kguttikondaevgenyl: its basically MAAS deployment failure;15:51
kguttikondaWhen I check in console I see following15:51
kguttikondawget -vvv --no-proxy --no-check-certificate --header=X-Bootaction-key: af96b250d0cbec284800fdb3457ba501e0747ac38ebaf8324ab5ca3db9fb9e66 https://drydock.ucp.svc.cluster.local:443/api/v1.0/bootactions/nodes/computer01o005/units15:51
kguttikondait shows connected and after that nothing; drydock lists the node as failure15:52
kguttikondawe have double checked the networking and from PXE interface we can reach the drydock and its connecting but not sure why its not downloading15:53
kguttikondabootdata15:55
kguttikondaThe boot data API is used by deploying nodes to load the appropriate boot actions to be instantiated on the node. It uses alternative authentication and is not accessible with Keystone.15:55
kguttikondaso, bootdata api do not use keystone auth; what will it use? how can we get X-Bootaction-ke?15:55
kguttikondaevgenyl: where as other nodes are able to download units and continue to download files15:57
evgenylkguttikonda: I would double-check the networking configuration in this case, check that the traffic from this node comes to the node with drydock-api (tcpdump), also check if drydock-api logs every request that comes to api and see if you can identify it.16:00
kguttikondathe node has only pxe interface; I thought by this time node should have bonding configured with OAM;16:01
kguttikondaI will deploy it once again and see if there is anything wrong with PCI bus and interface16:02
kguttikondaevgenyl: How can I do this?  check if drydock-api logs every request that comes to api and see if you can identify it.16:02
evgenylkguttikonda: you should be able to access the logs using something like`kubectl logs -f -n ucp drydock-api-UNIQUE-ID`16:04
kguttikondaevgenyl: Doesn't seems like it shows every transaction; I was already checking logs in both pods16:05
kguttikondaevgenyl: QQ: Can we use Airship to provision only kubernetes not OSH?16:19
evgenylkguttikonda: You can, you will need to remove OSH chart group from your site definition, here is an example for airsloop site https://github.com/airshipit/treasuremap/blob/master/type/sloop/manifests/full-site.yaml#L40-L5916:24
kguttikondaevgenyl: perfect; thank you16:24
kguttikondaalso; we can avoid bonding in hardware profile and host profile?16:25
evgenylkguttikonda: If your site is based on airship-seaworthy, you will need to add similar override as in airsloop, where you would remove unneeded charts.16:25
kguttikondaevgenyl: Got it16:25
evgenylkguttikonda: I'm not a huge expert in drydock/maas, but I think whatever you can do in maas, you can do in airship. This would require to edit hw profiles not to configure bonding, and also go through configs and fix interfaces names.16:26
kguttikondaOK; make sense; thanks evgenyl16:26
evgenylkguttikonda: You can refer to airsloop, which is as far as I can see requires only two interfaces, one pxe and another one for all other networks https://github.com/airshipit/treasuremap/blob/master/site/airsloop/profiles/host/compute.yaml#L25-L4116:33
kguttikondaevgenyl: cool, this will help16:34
kskelsexactly - airsloop has a config sample for drydock networks without the bond as well16:35
kskelshttps://github.com/airshipit/treasuremap/blob/master/site/airsloop/networks/physical/networks.yaml16:35
kskelsis basically the same without using bond16:35
kguttikondaOK; can we use airsloop with more computes then?16:36
kskelsso it really depends on your use ase16:36
kguttikondaok16:36
openstackgerritAlexander Hughes proposed airship/pegleg master: [DNM] Promenade genesis test  https://review.opendev.org/66338916:36
kskelsif you want to have full production-like deployment with HA control plane16:36
kskelsI would look into full airship-seaworthy16:36
kguttikondabasically; I have setup with number of servers but with 1 GB interfaces;16:36
kskelsif you are more willing to setup a lab/demo environment that would still have all the features but less dependencies and reqs16:37
kskelsthan airsloop is great way16:37
kskelsyes- it's very easy to add more compute nodes for airsloop16:37
kguttikondaoh Airsloop only have 1 controller; I get it16:37
kskelsyes16:37
kguttikondathen probably I need a hybird of Airsloop with 3 controller (Airship)16:37
kskelsyes - that's a good way to look at it16:38
kskelsI would base your things on airship-seaworthy16:38
kguttikondagoing to be an interesting work16:38
kskelsbut then mix and match some simplifications from airsloop16:38
kguttikondaI agree16:38
kskelsanother area will be disks16:38
kskelsairsloop is configured to work "out of the box" with 1 disk only16:38
kskelswhere else virrtual-seaworthy has nuymber of dedicated disks for ceph, etc16:38
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Adds unit tests for Spyglass CLI  https://review.opendev.org/66314016:39
kskelsin some ways - I would probably suggest to use airsloop as a start to get familiar with things16:39
kskelsand then move on to full HA16:39
kskelsbut def up to you and your use case16:39
openstackgerritAlexander Hughes proposed airship/pegleg master: [DNM] Promenade genesis test  https://review.opendev.org/66338916:40
kskels(e.g. airsloop was created for folks to have something more simple to get started)16:40
kskelsQQ: Can we use Airship to provision only kubernetes not OSH? - seeing your question before16:42
kskelsin this case probably nice to have k8s HA, etc16:42
kskelsso yeah - have fun!16:42
openstackgerritAlexander Hughes proposed airship/pegleg master: Pegleg dependency uplift  https://review.opendev.org/66310116:44
openstackgerritMerged airship/spyglass master: Schema validation  https://review.opendev.org/65917216:45
openstackgerritAlexander Hughes proposed airship/pegleg master: [DNM] Promenade genesis test  https://review.opendev.org/66338916:45
kguttikondathanks kskels16:52
openstackgerritKaspars Skels proposed airship/treasuremap master: Airsloop pipeline fixes  https://review.opendev.org/66345716:53
kguttikondaevgenyl: This time I did using update_site.sh script and the same compute node is able to download both units, files and completed deployment; Apparently there were no changes made to either network or node; This is crazy16:54
*** mrhillsman is now known as openlab17:02
*** openlab is now known as codebauss17:05
kguttikonda2019-06-06 17:08:29.273 8 INFO armada.api.middleware [-] POST http://armada-api.ucp.svc.cluster.local:8000/api/v1.0/apply?target_manifest=full-site&tiller_port=44134&tiller_host=10.97.181.254 - 409 Conflict17:10
kguttikonda2019-06-06 17:08:29.273 8 DEBUG armada.api.middleware [-] Response body:{"message": "Unable to acquire lock before timeout", "type": "error", "retry": false} process_response ./armada/api/middleware.py:14117:10
kguttikondaAny suggestions ?17:10
openstackgerritIan Pittwood proposed airship/spyglass master: Manifest undefined data validation  https://review.opendev.org/65568317:12
*** codebauss is now known as openlab17:13
*** openlab is now known as codebauss17:14
*** codebauss is now known as openlab17:15
evgenylkguttikonda: It may happen because you already have armada apply running somewhere.17:15
*** openlab is now known as codebauss17:16
kguttikondaevgenyl: last time when I ran deploy_site and I have stopped it using shipyard CLI17:16
evgenylkguttikonda: You can also try running something like `kubectl get locks --all-namespaces` and see if there is armada lock object, after that you can run describe to see what is the timestamp.17:16
evgenylkguttikonda: Oh, this must be it, I'm not sure if armada-api supports "stop".17:17
evgenylSo Shipyard API/AIrflow may have stopped, but Armada API may continue running the apply.17:17
kguttikondayeah, it was 2 hours back17:18
kguttikondaAPI Version:  armada.process/v117:18
kguttikondaData:17:18
kguttikonda  Last Updated:  2019-06-06T17:16:52Z17:18
kguttikondaKind:            Resource17:18
kguttikondaMetadata:17:18
kguttikonda  Creation Timestamp:  2019-06-06T15:51:46Z17:18
kguttikonda  Generation:          117:18
kguttikonda  Resource Version:    37033717:18
kguttikonda  Self Link:           /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock17:18
kguttikonda  UID:                 fce9d2f9-8872-11e9-b4f3-e4434b08312617:18
kguttikondaEvents:                <none>17:18
kguttikondawhat should I do now?17:18
kguttikondaShall I delete the lock in kube-system?17:18
evgenylkguttikonda: Can you check logs for both armada-api logs and see if any of those runs the deployment?17:18
kguttikondaupdate_site has been failed; but all nodes were deployed; So can I use just update_software.sh17:19
evgenylkguttikonda: You can delete it, but make sure that it is not being run by some of armada pods.17:19
kguttikondaI see armada-api logs nothing;17:19
evgenylkguttikonda: You can delete the lock and restart, if you are certain that none of armada instances runs the deployment.17:20
evgenylkguttikonda: You can run update software if you don't need to provision new nodes.17:20
kguttikondaok17:20
kguttikondayeah, in both armada-api pods I see no logs17:20
kguttikondaIn armada-api I see following17:21
kguttikonda2019-06-06 17:20:52.851 10 DEBUG armada.handlers.lock [-] Updating lock update_lock ./armada/handlers/lock.py:15517:21
kguttikonda2019-06-06 17:16:52.566 10 DEBUG armada.handlers.lock [-] Updating lock update_lock ./armada/handlers/lock.py:15517:21
kguttikonda2019-06-06 17:17:52.636 10 DEBUG armada.handlers.lock [-] Updating lock update_lock ./armada/handlers/lock.py:15517:21
kguttikonda2019-06-06 17:18:52.707 10 DEBUG armada.handlers.lock [-] Updating lock update_lock ./armada/handlers/lock.py:15517:21
kguttikonda2019-06-06 17:19:52.782 10 DEBUG armada.handlers.lock [-] Updating lock update_lock ./armada/handlers/lock.py:15517:21
evgenylkguttikonda: Oh, it means it still runs it.17:21
evgenylkguttikonda: It probably wait for some resource to be deployed.17:22
evgenylMaybe there is some job/pod that is not ready.17:22
kguttikondanot sure what exactly its trying to deploy17:22
*** chinna has quit IRC17:22
evgenylkguttikonda: If you scroll up the log, you should be able to see what was the last thing it applied.17:22
openstackgerritIan Pittwood proposed airship/spyglass master: Add tests into Zuul config and Makefile  https://review.opendev.org/66372917:23
*** codebauss is now known as mrhillsman17:24
openstackgerritIan Pittwood proposed airship/spyglass master: Add tests into Zuul config and Makefile  https://review.opendev.org/66372917:24
openstackgerritIan Pittwood proposed airship/spyglass master: Add tests into Zuul config and Makefile  https://review.opendev.org/66372917:25
kguttikondaevgenyl: i see it does ceph-rbd-pool17:27
kguttikonda2019-06-06 15:53:11.861 10 DEBUG armada.handlers.wait [-] Resource ceph-rbd-pool is ready! handle_resource ./armada/handlers/wait.py:18217:27
kguttikondabut I can see all pods are running fine in tenant-ceph17:27
evgenylCan you check that all jobs finished the execution? You can also run `helm list -a` to see if there are any non-deployed charts.17:28
evgenylkguttikonda: first can be checked by running `kubectl get jobs --all-namespaces`17:28
evgenylThere should be some kind of "success counters" in the columns.17:29
kguttikondaI see one of ceph-mon in tenant-ceph is being restarted17:29
kguttikondathat's the problem17:29
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary definition  https://review.opendev.org/65795317:30
*** mnaser has left #airshipit17:34
kguttikondaevgenyl: jobs are fine; I had the delete the ceph-mon data in one controller node and now ceph-mon pods are running and active; However no success in Armada-api17:35
kskelsthe armada takes a while to fail17:37
kskelsso likely existing shipyard command is still running - you can see it by checking get actions from shipyard CLI17:37
kskelsthere is no really good way to abort ongoing action - so you just need to wait out17:37
kskelsonce it's completed/timedout - you should be able to run update_site - e.g. armada will be able to aquire lock17:38
kskelsin other words - another run of armada is likely happening that is blocking your new task17:38
kguttikondakskels: there is no shipyard action running for sure; But stopping shipyard action doesn't seem to stop armada once it handover17:42
kguttikondaI just did ceph reweight OSDs and now armada moved on17:43
kguttikonda:)17:43
kskelsyes - armada will re-try17:43
kskelsso as long as you push in new docs - next re-try will take the new set17:43
kskelsright17:43
kskelsstopping action would not stop armada17:43
kskelsyou still need to wait for it to time-out17:44
kguttikondaok17:44
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Update dockerfiles to include ssh client  https://review.opendev.org/66373217:46
openstackgerritIan Pittwood proposed airship/spyglass master: Add tests into Zuul config and Makefile  https://review.opendev.org/66372917:53
openstackgerritAlexander Hughes proposed airship/pegleg master: Pegleg dependency uplift  https://review.opendev.org/66310117:54
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Update dockerfiles to include ssh client  https://review.opendev.org/66373217:54
openstackgerritIan Pittwood proposed airship/spyglass master: Add tests into Zuul config and Makefile  https://review.opendev.org/66372917:54
openstackgerritAlexander Hughes proposed airship/pegleg master: [DNM] Promenade genesis test  https://review.opendev.org/66338917:54
alexanderhughesgood lord with my proposals.  wonder if we should edit the notifications to not include WIP or DNM patchsets17:57
*** michael-beaver has joined #airshipit18:02
*** roman_g has quit IRC18:04
openstackgerritMerged airship/promenade master: New source for hyperkube binary definition  https://review.opendev.org/65795318:05
openstackgerritMerged airship/airship-in-a-bottle master: Support for new hyperkube source in promenade  https://review.opendev.org/65795418:09
dwaltalexanderhughes: I think we can only filter events18:10
dwalti.e. change merged, patchset created18:10
*** AlexNoskov has joined #airshipit18:11
alexanderhughesyeah I don't think it's a huge deal in the grand scheme of things, just noticed it and was a bit amused18:11
openstackgerritIan Pittwood proposed airship/spyglass master: Adds unit tests for Spyglass CLI  https://review.opendev.org/66314018:48
*** kguttikonda has quit IRC18:51
*** AOjeaGarcia has quit IRC19:03
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Update dockerfiles to include ssh client  https://review.opendev.org/66373219:15
openstackgerritIan Pittwood proposed airship/spyglass master: Adds unit tests for Spyglass CLI  https://review.opendev.org/66314019:22
alexanderhughescan I get a core to review https://review.opendev.org/#/c/645017/ please?  Lev left so I took over his changes.  not comfortable with +2ing something I was the last person making changes to19:34
openstackgerritMichael Beaver proposed airship/shipyard master: [WIP] Add override-checkout to OSH-infra  https://review.opendev.org/66228419:43
openstackgerritMerged airship/promenade master: templates: separate genesis and join sources  https://review.opendev.org/65842119:56
openstackgerritMichael Beaver proposed airship/shipyard master: [WIP] Add override-checkout to OSH-infra  https://review.opendev.org/66228420:09
*** kranthikirang has joined #airshipit20:11
openstackgerritRahul Khiyani proposed airship/promenade master: [WIP]ETCD: Add pod/container security context  https://review.opendev.org/66233420:18
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Update dockerfiles to include ssh client  https://review.opendev.org/66373220:20
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Update dockerfiles to include ssh client  https://review.opendev.org/66373220:20
openstackgerritEvgeniy L proposed airship/treasuremap master: Uplift nfs-provisioner  https://review.opendev.org/66258320:33
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Update dockerfiles to include ssh client  https://review.opendev.org/66373220:41
openstackgerritMerged airship/spyglass-plugin-xls master: Adds Safety dependency vulnerability checks  https://review.opendev.org/66203920:55
*** henriqueof has joined #airshipit20:58
hogepodgemattmceuen: did we move the sync meeting? On the etherpad it said Friday but my invitation still says Thursday20:59
*** MattCoachCarter has quit IRC21:00
openstackgerritAlexander Hughes proposed airship/pegleg master: Update dockerfiles to include ssh client  https://review.opendev.org/66373221:16
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: [WIP] Add unit tests for xls plugin CLI  https://review.opendev.org/66314221:17
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Site and global secret decrypt.  https://review.opendev.org/66284721:17
*** KeithMnemonic has joined #airshipit21:17
openstackgerritMerged airship/pegleg master: Adds Safety dependency vulnerability checks  https://review.opendev.org/66281221:19
*** kranthikirang has quit IRC21:19
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Updates YAPF settings to match Spyglass  https://review.opendev.org/66202021:23
*** howell has quit IRC21:26
*** alexanderhughes has quit IRC21:43
openstackgerritDan Crank proposed airship/promenade master: [WIP] Fixes to etcd backup script  https://review.opendev.org/66378522:19
openstackgerritAlexander Noskov proposed airship/treasuremap master: Uplift all components except Armada  https://review.opendev.org/66208522:27
openstackgerritAlexander Noskov proposed airship/treasuremap master: Uplift all components except Armada  https://review.opendev.org/66208523:31
*** nishantkr has quit IRC23:45
*** aaronsheffield has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!