At AllCode, we’ve learned a few lessons after running production environments for 10 years.
-
- Never roll out to production on Friday - When you’re rolling out on Friday, the rollout will always get delayed to Friday afternoon. Something will go wrong. No one wants to work over the weekend due to a decision made to roll out on Friday.
- Never run a database on a virtual machine when an automated database solution is available. If you run a database on a virtual machine, you won’t make the IOPS better than on the managed database. You won’t implement a proper backup strategy. When given the option, always run the database on RDS instead of EC2.
- When rolling out to production, always make sure that you take a backup of the production database instance before performing the rollout. We’ve seen several rollouts go awry because the database migrations fail, and the migration rollback has never been tested.
- When rolling out to production, always make sure that you have another instance running to point to if something goes wrong. It doesn’t even have to be a production version.
- Adopt Continuous Integration and Continuous Delivery - Sometimes your clients may not want to pay for it. Do it anyway to maintain your sanity.
- Automate Everything - From testing to deployment to rollback.
Joel GarciaJoel Garcia has been building AllCode since 2015. He’s an innovative, hands-on executive with a proven record of designing, developing, and operating Software-as-a-Service (SaaS), mobile, and desktop solutions. Joel has expertise in HealthTech, VoIP, and cloud-based solutions. Joel has experience scaling multiple start-ups for successful exits to IMS Health and Golden Gate Capital, as well as working at mature, industry-leading software companies. He’s held executive engineering positions in San Francisco at TidalWave, LittleCast, Self Health Network, LiveVox acquired by Golden Gate Capital, and Med-Vantage acquired by IMS Health.
