Photon Controller .9 in a lab!

I’ve been playing around with Photon Controller since .8 and I’m impressed by it’s potential.  VMware has a strong cadence for Photon Controller as .9 dropped a little more than a month after .8.

I’m sharing my experiences with running Photon Controller in a lab.

First some info on the .9 release:

• The OVA for the deployer is 3x smaller than .8 in size!
• Support for Kube-up is included.   This required changes on the Kubernetes side as well.  These changes are currently in master on the Kubernetes project.
• Docker Machine support!
• Production ready Bosh CPI

Get the bits here!

Now for the magic!

My Photon lab is a group of nested ESXI hosts running on a single host with 8 cores, 96 GB of memory and a 256GB SSD.  I’m using 1 node out of my C6100 to run the lab.  There has been a drop in memory prices recently so I doubled the memory in my lab.

That being said, you can install on far less memory and here is how!

I run Photon Platform on nested ESX and the performance is exceptional for a lab.  I’m running a total of four nested hosts.  One host for management and three hosts as cloud nodes for workload.  My hosts are each set to 18GB of memory. However, you can  deploy Photon with K8S using less than 4GB per host (16GB total) if you really press.

1. Install the Photon deployer on your physical host or laptop.  The deployer must  be network accessible to the ESX hosts where photon controller will be deployed.
2. Create Nested ESX hosts for the Photon Controller management and cloud nodes.  Each host will require a minimum of 4GB of memory.  I personally run with 18GB due to my physical hosts memory size.
   • Get the nested ESX OVA by William Lam and install it on your ESX host.
     1. Ensure that promiscuous mode and forged transmits are enabled on the virtual switch on the physical ESX host.
     2. Ensure that MAC learning dvFilter is enabled on the nested hosts.
   • Create a shared VMDK.  Your hosts will all use this VMDK as their image datastore to share images across all the hosts.
     1. Ensure that the VMDK is created as thick eager zero. I’m utilizing less than 1GB in my lab with a K8S installation. Adjust the size if you plan on loading more images.
     2. Ensure that the VMDK is enabled for mult-write
   • Create a local datatstore for each host.
     1. The local datastores can be created as thin
     2. The local datastores will be used for running the VMs, so they should be a  larger.  I’m currently just under 10GB utilization for the local datastores.
     3. Note that you can add local datastores later if more storage is required.  Photon controller will simply begin utilizing the additional datastores.

3.  Modify the Photon Controller deployment file to minimize management memory.  I’ve included my YAML deployment file below.

hosts:
  – metadata:
      MANAGEMENT_VM_CPU_COUNT_OVERWRITE: 2
      MANAGEMENT_VM_MEMORY_MB_OVERWRITE: 4096
      MANAGEMENT_VM_DISK_GB_OVERWRITE: 80
      MANAGEMENT_DATASTORE: esx-pp1-local
      MANAGEMENT_PORTGROUP: VM Network
      MANAGEMENT_NETWORK_NETMASK: XX.XX.XX.XX
      MANAGEMENT_NETWORK_DNS_SERVER: XX.XX.XX.XX
      MANAGEMENT_NETWORK_GATEWAY: XX.XX.XX.XX
      MANAGEMENT_VM_IPS: XX.XX.XX.XX
    address_ranges: XX.XX.XX.XX
    username: root
    password: XXXXXXX
    usage_tags:
      – MGMT
  – address_ranges: XX.XX.XX.XX-XX.XX.XX.XX
    username: root
    password: password
    usage_tags:
      – CLOUD
deployment:
  resume_system: true
  image_datastores: cloud-store
  auth_enabled: false
  stats_enabled: false
  use_image_datastore_for_vms: true
  loadbalancer_enabled: true

Note that the following from the deployment file.

•  I added the following to restrict the size of the management VM. Note the memory is 4GB, you could deploy with 2GB to further compress the size.

 MANAGEMENT_VM_CPU_COUNT_OVERWRITE: 2
 MANAGEMENT_VM_MEMORY_MB_OVERWRITE: 4096
 MANAGEMENT_VM_DISK_GB_OVERWRITE: 80

• The management datastore is specified as the following.

 MANAGEMENT_DATASTORE: esx-pp1-local

Note this is a local disk on the host named as a management host. My config only uses 1 host for management.  If you have more than one host, then this datastore must be a shared datastore with all management hosts attached.

• The shared image store to be connected to all hosts is specified as

image_datastores: cloud-store

Note that  “use_image_datastore_for_vms: true”  allows the image datastore to be used for deploying VMs too.  You can set this to false if you have other disks attached to be used for the VMs.

•  The IP address range property sets the number of cloud hosts to be used. In this case I use an IP range that spans three IP addresses.

address_ranges: XX.XX.XX.XX-XX.XX.XX.XX

4.  Modify the flavors to minimize the K8, Mesos or Swarm deployments.  Please read my earlier post or William Lam’s blog on how to set the flavors.

Script the install with Photon CLI

Within my lab I have the install of Photon Controller scripted with  photon CLI. A new tenant that includes a K8S project is ready for use in under 10 minutes.  I’ve included the script below.  Note: WordPress treats certain characters like ” and — funny. You may have to fix these characters.

#!/bin/bash

# This script assumes a fresh Photon Controller deployment and is for demonstration purposes.
# The script will create a tenant, resource ticket, project and a K8S cluster

P2_DEPLOYER="http://XX.XX.XX.XX"
P2="http://XX.XX.XX.XX:28080"
TENANT="inkysea"
TICKET="k8s-ticket"
PROJECT="go-scale"

# Deploy Photon Controller
photon -n target set $P2_DEPLOYER
photon -n system deploy esxcloud-installation-export-config-CLI.yaml


photon -n target set $P2

photon -n tenant create $TENANT
photon tenant set $TENANT
photon -n resource-ticket create --name $TICKET \
    --limits "vm.cpu 24 COUNT, vm.memory 64 GB, vm 100 COUNT,ephemeral-disk.capacity 2000 GB,ephemeral-disk 12 COUNT"
photon -n project create --resource-ticket $TICKET --name $PROJECT \
    --limits "vm.cpu 8 COUNT, vm.memory 38 GB, vm 25 COUNT,ephemeral-disk.capacity 440 GB, ephemeral-disk 4 COUNT"
photon project set $PROJECT


# Deploy k8s
PHOTON_ID=`photon -n deployment list | awk '{if (NR!=1) {print}}'`
echo "Photon Controller Deployment ID is $PHOTON_ID"

IMG_ID=`photon -n image create photon-kubernetes-vm-disk1.vmdk -n photon-kubernetes-vm.vmdk -i EAGER `
#echo "Kubernetes Image ID is $IMG_ID"

photon -n deployment enable-cluster-type $PHOTON_ID -k KUBERNETES -i $IMG_ID

photon cluster create -n k8-cluster -k KUBERNETES --dns XX.XX.XX.XX --gateway XX.XX.XX.XX --netmask 255.255.255.0 \
    --master-ip XX.XX.XX.XX --container-network 10.2.0.0/16 --etcd1 XX.XX.XX.XX -s 1