{"id":498,"date":"2016-06-07T15:57:32","date_gmt":"2016-06-07T14:57:32","guid":{"rendered":"http:\/\/oso.com.pl\/?p=498"},"modified":"2016-06-07T08:57:57","modified_gmt":"2016-06-07T07:57:57","slug":"things-that-keep-me-awake-at-night-devops-pipeline-quality-assurance","status":"publish","type":"post","link":"https:\/\/oso.com.pl\/?p=498&lang=en","title":{"rendered":"Things that keep me awake at night &#8211; DevOps pipeline Quality Assurance"},"content":{"rendered":"<p>Gathering\u00a0thoughts as I organize this topic, and explore it.\u00a0Suggestions on where to learn more are welcome.<\/p>\n<p>I&#8217;ll be simplifying things initially. I plan to start by focusing on the pipeline and tools used in the DevOps culture. I expect the more I dive in this topic, the more precise next posts will be.<\/p>\n<p>For simplicity\u00a0let&#8217;s assume we are dealing with development of a cloud-based web application, having a DevOps toolchain that includes tools like Chef, Puppet, Jenkins, Docker, Packer, AWS, New Relic, Splunk&#8230; how do you test a deployment pipeline built on top of these?<\/p>\n<p>I have to start somewhere. I know this: you can approach testing software\u00a0by dividing the problem into separate areas, researching them, and executing any necessary actions, including finding and resolving issues. The result should hopefully be a\u00a0high quality, or at least acceptable, product.<\/p>\n<p>Let me try applying these areas to DevOps toolchain, and list the questions\/topics that emerge.<\/p>\n<h5>Functional<\/h5>\n<ul>\n<li>Is it working as expected? What does working as expected mean to you? To your stakeholders?<\/li>\n<li>Do you have unit tests? Integration? End-to-end? How many is enough?<\/li>\n<li>Do you need to do any manual testing after a pipeline step is executed?<\/li>\n<\/ul>\n<h5>Automation \/ Automatability \/ Testability<\/h5>\n<ul>\n<li>Are you going to automate the testing? Why yes? Why not? How much?<\/li>\n<li>If yes &#8211; which tools will you use? Are they free? What alternatives do you have?<\/li>\n<li>Is the toolchain automation-friendly? Was it created with automation in mind?<\/li>\n<li>Is it testing-friendly in general? Do you have hooks \/ breakpoints to make it easy to test?<\/li>\n<\/ul>\n<h5>UI\/UX<\/h5>\n<ul>\n<li>Is there a certain\u00a0User Experience your DevOps tools should deliver?<\/li>\n<li>Is the pipeline error-prone? Can somebody deploy a test build to production by mistake? Can they destroy your current production stack by clicking on a badly described button?<\/li>\n<li>Do you need to support keyboard shortcuts? Arrow keys \/ tabs to navigate?<\/li>\n<li>Does the\u00a0UI support long\/short inputs for build names, or components? High build numbers?<\/li>\n<\/ul>\n<h5>User Acceptance<\/h5>\n<ul>\n<li>Who is your customer? What acceptance do you need from them?<\/li>\n<li>Would you do A\/B testing for your pipeline?<\/li>\n<\/ul>\n<h5>Installation \/ Integration \/ System<\/h5>\n<ul>\n<li>What do you need to integrate with? For example &#8211; would you file JIRA tickets automatically if something goes wrong?<\/li>\n<li>Do\u00a0you need a database? Which version?<\/li>\n<li>What operating system will your toolchain run on? What OS will you support for developing it?<\/li>\n<li>When depending on a 3rd party &#8211;\u00a0do you accept to rely on their uptime? What if critical cloud-based tool goes down when you urgently need to deploy a hotfix?<\/li>\n<li>Will the 3rd party\u00a0let you know\u00a0of planned downtime? Is the downtime in a timezone suitable for you?<\/li>\n<li>Do you have backup?<\/li>\n<\/ul>\n<h5>Compatibility<\/h5>\n<ul>\n<li>What platforms should you be compatible with? AWS? OpenStack? Azure? Are you going to test all of them?<\/li>\n<li>If your pipeline is web-based &#8211; which browsers will you support? Can a bad rendering on Safari cause an error? What about strict Firefox security? What if the users are running Chrome with JS-blocking extension?<\/li>\n<li>Any potential compatibility issues between your tools? Should you test every new version with others?<\/li>\n<\/ul>\n<h5>Globalization<\/h5>\n<ul>\n<li>Do you have any dates or numbers showing up in the pipeline? 1.000 and 1,000 are not the same&#8230; same goes for 6\/12\/2016&#8230;<\/li>\n<li>Monday is not the first day of the week for everybody. Do you care?<\/li>\n<li>If you have user input &#8211; does it support non-ASCII characters? Does it have to?<\/li>\n<li>Any of your users need a localized UI?<\/li>\n<li>If some of your resources are outside your country &#8211; would you support them? What if part of the deployment needs a phone number, but it&#8217;s in a weird formatting from another country?<\/li>\n<\/ul>\n<h5>Compliance<\/h5>\n<ul>\n<li>Are you required to meet certain requirements like SOX or HIPAA?\u00a0Can your DevOps toolchain and code assure at least part of the compliance?<\/li>\n<li>Any export regulations you might be violating with your DevOps code? What if certain country requires that data is stored locally, but your tools deploy a server on a different continent?<\/li>\n<\/ul>\n<h5>Stress \/ Load \/ Performance<\/h5>\n<ul>\n<li>Can you deploy 10 servers simultaneously? What about 10000?<\/li>\n<li>How long does it take to deploy the infrastructure? Is 1 hour acceptable? What if 10 minutes is too long?<\/li>\n<li>Did anyone even define these requirements?<\/li>\n<li>Do you track\u00a0any of the performance metrics?<\/li>\n<\/ul>\n<h5>Security<\/h5>\n<ul>\n<li>Do you take any user input? Can a malicious user infect other users? Steal their passwords? Admin password?<\/li>\n<li>Do you store sensitive data in your Jenkins jobs? Where do you store them securely?<\/li>\n<li>How will you prevent users from committing their AWS credentials to public repositories?<\/li>\n<li>Do you remove all access when terminating employees?<\/li>\n<li>Do you use access control? Do you audit user actions? Should you?<\/li>\n<li>Who is really implementing security? Can a single engineer misconfigure firewall on all your production servers?<\/li>\n<\/ul>\n<h5>Supportability<\/h5>\n<ul>\n<li>Do you have enough logging to know why something went wrong? Do all\u00a03rd party tools have enough logging?<\/li>\n<li>Where are\u00a0your logs?<\/li>\n<li>Do you have alerts \/ notifications in place?<\/li>\n<\/ul>\n<h5>Configuration<\/h5>\n<ul>\n<li>What are the configuration options for your jobs?<\/li>\n<\/ul>\n<h5>Documentation<\/h5>\n<ul>\n<li>What documentation do you need? Do you have enough if somebody decides to leave abruptly or falls under a bus?<\/li>\n<li>Any public-facing documentation you want to \/ have to share?<\/li>\n<\/ul>\n<h5>Adoption \/ Metrics and Instrumentation<\/h5>\n<ul>\n<li>Any metrics you want to track?<\/li>\n<li>Do you need to add instrumentation to the jobs to know where the bottlenecks are?<\/li>\n<\/ul>\n<h5>Upgrade \/ Rollback<\/h5>\n<ul>\n<li>How will you test new versions of the tools? Are you ready to roll them back? Will they work after rollback?<\/li>\n<\/ul>\n<h5>Rollout strategy<\/h5>\n<ul>\n<li>What is your must-have vs nice-to-have? What tools depend on each other?<\/li>\n<li>Can you define phases of your DevOps toolchain deployment?<\/li>\n<\/ul>\n<h5>Resources<\/h5>\n<ul>\n<li>Have you identified all the resources you need for testing?<\/li>\n<li>Environmental resources like hardware, and software that you need?<\/li>\n<li>What about licenses? Any legal review of these needed?<\/li>\n<li>Are you well staffed? Any training your engineers need?<\/li>\n<\/ul>\n<h5>Deliverables<\/h5>\n<ul>\n<li>Documentation, artifacts&#8230; what else do you need to deliver?<\/li>\n<\/ul>\n<h5>Vendor \/ 3rd party<\/h5>\n<ul>\n<li>When working with a vendor on your DevOps implementation: how much would you want them to test vs you? What is their testing strategy? How much testing overlap should happen? What to they need to deliver?<\/li>\n<\/ul>\n<h5>Definition of done<\/h5>\n<ul>\n<li>When can you tell you are happy with the testing of the DevOps toolchain?<\/li>\n<li>Do you need to sign off? Who else signs off?<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>These are just some initial thoughts. What do you think of these? What&#8217;s missing?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Gathering\u00a0thoughts as I organize this topic, and explore it.\u00a0Suggestions on where to learn more are welcome. I&#8217;ll be simplifying things initially. I plan to start by focusing on the pipeline and tools used in the DevOps culture. I expect the more I dive in this topic, the more precise next posts will be. For simplicity\u00a0let&#8217;s &hellip; <a href=\"https:\/\/oso.com.pl\/?p=498&#038;lang=en\" class=\"more-link\">Czytaj dalej<span class=\"screen-reader-text\"> \u201eThings that keep me awake at night &#8211; DevOps pipeline Quality Assurance\u201d<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[20],"tags":[27],"class_list":["post-498","post","type-post","status-publish","format-standard","hentry","category-testing","tag-devops"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p217OK-82","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/oso.com.pl\/index.php?rest_route=\/wp\/v2\/posts\/498","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oso.com.pl\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/oso.com.pl\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/oso.com.pl\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/oso.com.pl\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=498"}],"version-history":[{"count":13,"href":"https:\/\/oso.com.pl\/index.php?rest_route=\/wp\/v2\/posts\/498\/revisions"}],"predecessor-version":[{"id":511,"href":"https:\/\/oso.com.pl\/index.php?rest_route=\/wp\/v2\/posts\/498\/revisions\/511"}],"wp:attachment":[{"href":"https:\/\/oso.com.pl\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=498"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/oso.com.pl\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=498"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/oso.com.pl\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=498"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}