At AllCode, we’ve learned a few lessons after running production environments for 10 years.
-
- Never roll out to production on Friday - When you’re rolling out on Friday, the rollout will always get delayed to Friday afternoon. Something will go wrong. No one wants to work over the weekend due to a decision made to roll out on Friday.
- Never run a database on a virtual machine when an automated database solution is available. If you run a database on a virtual machine, you won’t make the IOPS better than on the managed database. You won’t implement a proper backup strategy. When given the option, always run the database on RDS instead of EC2.
- When rolling out to production, always make sure that you take a backup of the production database instance before performing the rollout. We’ve seen several rollouts go awry because the database migrations fail, and the migration rollback has never been tested.
- When rolling out to production, always make sure that you have another instance running to point to if something goes wrong. It doesn’t even have to be a production version.
- Adopt Continuous Integration and Continuous Delivery - Sometimes your clients may not want to pay for it. Do it anyway to maintain your sanity.
- Automate Everything - From testing to deployment to rollback.
