The Jay Nine Inc Development Pipeline Explained

Jay Nine, Incorporated utilizes a development process heavily based on DevOps and “Continuous Integration” principles. We’ve found that an easy way to boost the productivity and quality of our business app development is by utilizing continuous integration.

Continuous integration provides safeguards for teams—even across different time zones—to create large and complex systems with a drastically higher level of confidence and control. The apps those teams build and deploy are more efficient, providing rapid feedback on any problems we may introduce with the changes we commit. says “The key goals of continuous integration are to find and address bugs quicker, improve software quality, and reduce the time it takes to validate and release new software updates.”

In essence, we want a fast flow of left-to-right from development to IT operations to the end user (customer). To maximize this flow, we need to bring small pieces of a project to production faster, using more efficient intervals of work.

For the sake of explanation, I’m referring to everything as a “task.” A task can be as small as changing the logo on a website, or as large as creating a new app from scratch.

The Deployment Pipeline

The pipeline for taking a task from production to development should be:

  1. Analyze the Requirements
    1. Seek perfect clarity on what the task at hand is
    2. Take a minute and make sure that this is a task that needs to be done properly in the first place. There is nothing worse than doing well that which not be done at all.
    3. Confirm all developers fully understand the requirements
    4. Create the test case we will run to test the task
  2. Design/Programming
    1. Task is developed and worked on
    2. Task is documented
    3. The process for testing this task is effectively passed down to QA
  3. Implementation
    1. Task is moved to the testing environment
  4. Testing/Verification
    1. QA should review the document of tests created in step 1 or step 2
    2. QA should test the task
    3. QA needs to test the task to try and break it (as a user will) not to try and make it “pass”
    4. If the test passes, it can be moved into production. QA needs to use a video to document testing the test, one that can be sent to pertinent parties to confirm the success or failure of the task
    5. If the test fails, it should go back to step 1, and repeat

The Pipeline Dissected

The first step is to “Analyze the Requirements.

We need to be sure that this is a task that needs to be done (not just some fleeting idea or “nice-to-have” that is distracting from more important tasks). The developers and builders understand with perfect clarity what the final product looks like. If they don’t, they need to.

For example, if we’re switching out the logo on a website, the test would be:  “Logo should look proper on all major devices (including phones) with no pixelation and without impacting the page load negatively.”

The philosophy dictates that the teamn needs to develop the test, first. This is a process known test-driven development. First, the developer writes a failing test, and then completes the task until the test passes. If the test doesn’t pass, the developer knows the task isn’t ready yet

The “Design/Programming” step is a fairly straightforward second step. The programmer develops the code for this task until the test written in the prior version passes. The biggest failure point here is when a junior level developer goes unchecked in creating the code for the task. Remember, it’s fundamental the task is thought of in the ecosystem of the application as a whole. In the example of updating the logo on a website, if the programmer isn’t careful they can break the elements around the logo (often either the website tabs/navigation or the phone number/contact information). The programmer needs to ensure this is done to be scalable and not break important other functionality.

Next is the “Implementation” step, where the program moves to a “staging” version of the app. This environment is a replica of the live server.

The last step is “Testing/Verification.” In this stage, a user or QA person uses the task description provided by the business, and make sure it works as intended. The goal here is not to ensure the task works.

Instead, the goal is to try and break the program.

If a QA person isn’t ruthlessly trying to break the implemented task, we can be sure the users will. Completion of this task should include an easy demo via video (which is better when working across time zones or busy schedules) and any necessary documentation needed to log a completed task.

Why is This Pipeline Important?

Fundamentally, this process must flow from left to right, and the team may find themselves spending more time than might be considered necessary on “Analyzing the Requirements.”

The environment these tasks are developed, tested, and deployed in is critical. All developers must be working in harmony to ensure full understanding of a task before it begins. There is no room for lazily investigating a task, nor room for cowboy programmers and working in silos.

All members of the team should be thinking about the project as a whole, not just some little part of the code. Identical to an assembly line, having to go back and fix a step doesn’t speed things up. It slows down production and deeply impacts the company as a whole. Here is a why, in the first stages of the process, developing tests and focusing on the end goal will minimize this downtime.

The business leaders and project managers need to question the validity of a task constantly. The leaders need to ensure the tasks are being done to benefit the end user, the final consumer of the program. There is nothing worse than doing well, that which need not be done at all.

Repetition and practice are the key steps to mastery. When we integrate this process, we require all developers to do this on even the most simple of tasks. Otherwise, it will fail in production. If the developers, QA teams, operation teams, and business executives aren’t working closely, it will exponentially increase the number of issues in the production software.

If Apple has taught users one thing, it’s that software should just work. There shouldn’t be a lot of unnecessary troubleshooting or education, a user wants the software to perform the task as they see it. The process outlined in this article helps ensure that we can deploy many tasks into production using continuous deployment principles while giving users exactly what they want (increasing the profitability of the software).

What is DevOps? How Does DevOps Improve Software Development and IT Projects?

what is devops

DevOps (a play on “Development” and “Operations) is an enterprise software development concept meant to join Development and IT operations with improved communication and collaboration between these two business units.

To the outside looking in, “IT” generally encompasses all areas related to technology, be it programmers, the guy who fixes your printer, or the guy who maintains your current systems. To the non-technical types, there is a myth that “being good with computers” means you can do any field of computing expertise.

In the medical field, if you have a vitreoretinal illness that requires a specialized surgery, you’re going to contact an ophthalmologist–not an orthopedic doctor. While the latter will likely do better than say, myself, you want someone who has studied that particular field of medicine and does it all day.

DevOps is a system that helps merge these different IT fields together, specifically focused on making sure the people who will maintain and use the application are involved in the process from start to finish. DevOps then goes a step further, with a focus on bringing an entire organization together to ship better software, more frequently.

Under a DevOps model, development and operations are no longer operating in echo-chambers and siloed. These two teams are merged together to allow engineers and different layers of expertise to work together during the entire application lifecycle, instead of just their core expertise.

The Three Ways: Principles of DevOps

DevOps is based on “Three Ways” of which all observed patterns and behaviors are structured. Unlike a more rigid system like Scrum, DevOps is much more philosophical:

From The DevOps Handbook:

The First Way enables fast left-to-right flow of work from Development to Operations to the customer. In order to maximize flow, we need to make work visible, reduce our batch sizes and intervals of work, build in quality by preventing defects from being passed to downstream work centers, and constantly optimize for the global goals.

The First Way is based on Agile/Lean production and focuses on moving bite sizes of the final product into production sooner rather than later while bringing all key players into the development cycle (from developers to project managers, and finally–the customer).

The Second Way enables the fast and constant flow of feedback from right to left at all stages of our value stream. It requires that we amplify feedback to prevent problems from happening again, or enable faster detection and recovery.

The Third Way enables the creation of a generative, high-trust culture that supports dynamic, disciplined, and scientific approach to experimentation and risk-tasking, facilitating the creation of organizational learning, both from our successes and failures. Furthermore, by continually shortening and amplifying our feedback loops, we create ever safer systems of work and are better able to take risks and perform experiments that help us learn faster than our competition and win in the marketplace.

The Second Way & Third Way are focused on creating safer systems of work where problems can be discovered and fixed without having to go through a committee.

Similar to the leader-leader method developed by Captain Marquet. Captain Marquet was a Navy captain, responsible for developing a unique method of leadership after he gave an impossible order to his men one day.

His method developed around the philosophy of never giving another order, rather, giving an intent. So, instead of (moving back to web development) a project manager saying “create a shopping cart with products for this new website” it’s “the intent here is to create a shopping cart that women in the late 40s, affluent, can use to purchase specialized spices for their kitchen.”

Of course, this doesn’t mean that everything is left to the discretion of the developer and lower-on-the-totem-pole people, the rules of the system (i.e., the generated PDFs from this application have to look exactly like our other branded materials, or, the conversion tracking on a website needs to be functioning 100% of the time) still apply.

What is removed is the ego of the project manager, company owner, client, and other factors that have a habit of delaying projects and causing confusion. This allows for all to have a core focus on the rules of the system.

How Does DevOps Work in an Organization?

Software and the Internet have helped transform this world, and software no longer merely supports a business; it is now a fundamental aspect of every business.

As Gene Kim (one of the founders of DevOps) mentions in a Forbes article:

“…which tended to be a set of principles from which one can derive all of the observed DevOps patterns. The first is all about accelerating flow as you go from left to right, [Development] to [Operations], in the value stream. The second is the flow of feedback, how do you create effective feedback from Ops back to Dev so that when something goes wrong, you can either prevent it or detect it more quickly. The third is about creating a culture of continual learning and experimentation using the notions of high trust culture.”

In the age of digital business, the DevOps founders realized that sites with massive traffic like Google, Amazon, and Twitter, are all known for being able to do deployments (moving new features or bug fixes into production) many times a day (or even many times an hour). In order to deploy new ideas this quickly, you’re going to need a system that doesn’t break what’s already working, can be undone easily, and relies on open and honest communication with others within the organization.

As Amazon puts it:

In a similar way that physical goods companies transformed how they design, build, and deliver products using industrial automation throughout the 20th century, companies in today’s world must transform how they build and deliver software.

Working in a DevOps environment helps place the focus on constantly shipping products, features, and changes to the customers. It means that you can establish an environment where you can create feedback as quickly, cheaply, and rampantly as possible.

DevOps & Your Organization, a Few Troubles

Initially, especially for a simpler website build or smaller piece of business software, an organization may be turned off to paying for an environment of continuous deployments. A lot of companies like the idea of treating a software project like an architectural project or remodel: there is a beginning, a middle, and end–with a fixed cost and time line.

With DevOps, you’re constantly working on improving your software or product, and focused on shipping new changes and features in a rapid environment. This means that you either have staff members or an ongoing contract with a company to ensure that software is constantly being worked on, improved, and released.

Next, if DevOps is the main philosophy of an organization or project, c-suite executives, customers, and project managers begin to get spoiled and think that this means everything will be much faster to build and implement. This is rarely the case, complicated tasks will still take longer than “an afternoon” to complete.

If you’re going to completely change your application to move from the web to a desktop or iOS style application, DevOps will not get it done by the end of the day. Rather, it means that the philosophy of DevOps will be used to ensure that when you launch the application, you don’t spend the next 3 months putting out fires for angry customers and losing business.

The next issue we see a lot in implementing DevOps: dealing with egos. The human ego has a nasty way of creeping into processes and philosophies like this and losing sight of the core focus.

It is generally agreed that DevOps is about efficiently providing the customer with the best possible product. It’s not about providing the company CEO with a new website “look” two weeks after the last website was deployed because she found a new website she likes, and this is now the top priority.

It also doesn’t allow for inter-department wars, where programmers are furious because designers keep making “asinine changes” without seeing the full project picture or designer are furious that developers came back with a two week lead time on what they felt was a “simple fix.”

DevOps is all about breaking down isolation chambers, which can be extremely difficult for some organizations.

It’s Not Everything

Much like some of the other systems we’ve talked about, DevOps has attracted a lot of Zealots that exclaim this system can do everything and the kitchen sink. While DevOps is good at creating a collaborative and agile corporate culture, it is specific to how operations and development tie together to work in harmony. It should focus only on the delivery and project management of software, without trying to fit into areas it does not belong (like the overall company structure, human resources, or other similar items).

The Pros and Cons of Scrum

pros and cons of scrum

SCRUM is an Agile framework used for working on complicated projects, generally with a close-knit team. Simply, Scrum is a project management system.

A High-Level Overview of the SCRUM system:

  • Scrum consists of three distinct roles: product owner, scrum master, and team member
  • Scrum works by setting sequential goals that must be completed in a fixed length of time (called a “Sprint”)
  • First, the product owner creates a “backlog,” which is a fancy term for a wishlist of tasks that need to be prioritized on a project. This is the set of deliverables needed for the product.
  • Second, the Scrum team conducts a sprint planning session where parts of this wishlist are broken into smaller chunks.
  • Next, the team creates a sprint backlog, plans for the implementation, and settles on a time duration (usually 1 or 2 weeks) for every sprint
  • The team gets together every day for a scrum meeting where they share daily updates and access the progress of the project. This is led by the Scrum Master
  • The product owners and stakeholders review the sprint cycle at the end to assess progress

Pros Of Scrum

What Scrum does is bring teams together to create great things. Scrum requires all project members to see the end goal and to work incrementally towards it.

Scrum is focused on creating a workable product that can be brought to market as soon as possible.

Doing sprint cycles and focusing on fixed periods of time where you bite off small bits of the project is a welcome breath of fresh air in stale and non-moving projects. It focuses on making sure the product is deliverable, not that stakeholder’s egos are met.

Having a structured system is a great starting place for getting a project organized, on track, and meeting well thought-out objectives. The product backlog and user story is a great way to avoid scope creep and egocentric decision making.



SCRUM is one of those systems that really requires buy-in from all different aspects of the organization. A quote from Sutherland’s book on Scrum:

“For Scrum to really take off, someone in senior management needs to understand in his bones that impediments are nearly criminal.”

At first glance, it seems crazy to believe that anyone in senior management would accept and welcome impediments to any sort of project. This assumes two things, 1., that the impediment is obvious, and not an unforeseen 2., that ego isn’t involved.

I’m sure this doesn’t happen in your organization, but some organizations we’ve worked on have hired 1 – 3 designers and spent months tweaking a design until it was perfect. “Perfect” was only defined by the COO of the organization.

Scrum really requires a revolution of culture, which can be hard for businesses. It can be even harder if there is a larger disconnect between IT and the business teams.

Even if the business understands and buys into the Scrum process, individuals can have issues. This would be someone like a middle-level sales manager or analyst. Now, they have to make User Stories instead of the documents/mock-ups/spreadsheets they were used to.

I knew a developer who took over an extremely dysfunctional engineering team with about 100 people. The first thing he did was implement Scrum. It worked incredibly well for about 2 – 3 years.

Then, it didn’t work anymore. The team had matured, the organization had become much leaner, and the process of doing Scrum was just too much. It was time for a new change.

This is something a lot of Scrum and Agile advocates don’t realize or won’t admit — Scrum is just a tool for specific applications. Scrum is not a be-all-end-all system that will solve all of your organizational issues forever.

I’m reminded of a great quote in reference to Growth Hacker Marketing and Growth Hacking: “You can’t growth hack your way out of a PR crisis.” 

These lean and mean systems everyone is talking about work in dysfunctional environments where documentation is lack, communication is poor, and there seems to be no end in sight. The goal, I feel, should be to utilize pieces of these consistently, and only rely on the main systems in specific situations.

Like all systems, Scrum creates zealots too. It creates people who spend more time focusing on how Scrum is perfection, instead of seeing the pros and cons and focusing on the business objectives at hand.


Scrum, and similar systems have a habit of establishing the villainy of the status quo, and as such, a lot of the zealots spend more time refusing to see any downsides of Scrum, rather than looking at it as a useful tool for specific applications.

Another great quote in Sutherland’s book on Scrum:

“Years later it occurred to me that organizations, teams, and people are all complex adaptive systems. The same things that move cells from one state to another are also what move people from one state to another. To change a cell, you first inject energy into the system. At first, there’s chaos, there seem to be no rules, everything is in flux. When you do this to organizations trying to change, people freak out. They can’t understand what’s happening. They don’t know what to do. “

Overall I feel the principles behind Scrum are very solid. When executed properly, it really is a great way to get things done in a short period of time and to get an actual working product up and running.

Think a startup or major overhaul of an existing application. Scrum does a great job of focusing on the core components (what the users will use and need) and rolling out and testing those components. It has little room for the “surprise” meeting where we need to discuss a new offering because the CEO received a sales email from a competitor.

However, with an actual working product or existing project, I think elements of Scrum (the focus on sequential goals and eliminating waste) is a fantastic mindset all organizations should strongly push for. It’s the User stories, weekly/bi-weekly sprints, and similar that create issues.


How We Organize Our GIT and Why

organizing git

All of the projects for Jay Nine, Inc use a very specific process in with our development projects. Everything in the project must be checked into a single version control repository: all of the code, testing scripts, database scripts, build and deploy scripts, and anything else needed to create, install, run, and test your application.

This is for several reasons, namely:

  1. This makes it so that we have a ready-to-go back up of the entire application that we can launch at a moments notice
  2. When working with a team of developers, you can easily have multiple developers working on the project without a lot of hassle

If a bug is pushed live, our team has a rule that we try to fix it for ten minutes. If, after ten minutes, we don’t have a solution, we’ll roll back to the previous version in our version control and fix it from there.

This is auto-synced with our live websites, so when we make little fixes and update, checking in the completed code will automatically update the website (allowing for continuous deployment).

The GIT folders are organized very simply like this:

  1. The “Application” folder contains all of the necessary code that comprises the application
  2. The “Database” folder contains the latest copy of the database, in an executable format to create a new database if needed
  3. The “Documents” folder contains all of the project documents on how to use the project (end user manual) and how the project code is broken down
  4. The “Publishing and Testing Scripts” folder contains all of the documentation needed to publish the application in its environment, how to test the application to make sure no core functionality is broken, and a rollback plan in case you need to roll back for any reason

This has allowed us to drastically squash live bugs, as the checks and processes in place ensure things go live safely. A common issue with the continuous roll out plan and the frequent changes to working software is when you have live bugs.

Live bugs generally mean two things:

  1. Live bugs = lost money.
  2. Live bugs = people pulling their hair out because the customer facing version of their product/software is broken. What if a key customer sees this? What is a partner of ours sees this?

The version control system we use removes this fear. On top of that, it ensures there is always a working, ready-to-go version of the software in case something really bad happens.

The Jay Nine, Inc Software Development Process


We’re working on continuously improving our process of project management and development. The below outlines the high level overview of the processes we use to develop and build web applications and custom business software.

The Core Fundamentals:

  1. Find out exactly what the [user, client, or person who is benefiting the most from this system] needs and wants.
  2. Get the data you need to make that happen. Analyze at least ten types of forms. Refuse to accept any data without validating it first, always assume there is undiscovered data.
  3. Build the simplest and quickest version of that. Always focus on getting the simplest version to a solution in production as soon as possible.
  4. Test with user, fix mistakes, create better functionality. Ensure that no human error is found again (through automated tests and scripts).
  5. Do not allow arbitrary deadlines hurt long term productivity (focus on the tasks that will take a few extra days and save a few months). Set deadlines for each project on either weekly, every two weeks, or monthly deadlines. In emergencies, set daily deadlines. Agree upon this before the project begins.
  6. Ensure all changes are pushed to a staging environment, rigorously tested, and then pushed live after that.
  7. Ensure all people involved in the project have the tools and resources they need to see the project vision. The lowest developer or the highest project manager should be able to rollback changes if needed, or push new changes live easily.
  8. All project details are carefully, but not wastefully, documented.
  9. Nothing matters in the project other than working software, easy functionality, and results.

Programmers and developers are not the best at managing clients. Since we’re very literal people, we love systems that have precisely defined behaviors.

This makes working with non-technical people harder for us than it needs to be because we tend to take every request literally, rather than spending the time trying to figure out what a client actually wants.

The goals of our process system:

  1. To minimize bottlenecks, and spend as much time on the Code, Test, Deploy cycle as possible
  2. To create a system where even the entry level developers have the resources to make decisions and determine what the client actually wants, providing customers better systems and faster implementation cycles
  3. To measure all projects on weekly cycles to ensure that we are not losing focus on the big picture, but ensuring small tasks are taken care of efficiently

High-Level Planning Notes:

  1. To plan the project, we need access to all the data or a live test case. Not a promise of the data, not a vague understanding, but the exact items that we will be putting into production.
    1. This needs to be detailed enough to create an exact scope.
    2. We should have at least several different use cases/data examples to account for one-offs
    3. We always want to question the validity of the data and ensure we’re probing to iron out any outliers and cases that will break the system
  2. All development systems and task reporting needs to empower the developer, not add unnecessary work to the developer
    1. This means we need to streamline the process of a developer seeking higher level help, clarification, etc., without a lot of unnecessary reporting and systems
    2. Our motto: outsource anything we cannot do directly, automate anything we can automate, and eliminate anything that’s not crucial to the project
  3. All project tasks are updated in real-time in the task tracking (currently, Asana)
  4. Active documentation needs to be included in all parts of the process; all documentation needs to be up-to-date with no exceptions.
    1. Two “testing” documents are prepared and utilized for each client
      1. A document to use to guide testing
      2. A seperate document to use to track testing changes

Plan Asana Tasks for Web Development

As a rule of thumb, all tasks and documentation need to have necessary details. Usually an image/screenshot of the task, and text to explain the issue.

We organize each  project into with the main boards of:

  • Wishlist (a place where we can store dreams/wanted tasks that aren’t in production)
  • Bugs (bugs that are on the live site and would require pertinent attention)
  • In Progress (tasks that are currently being worked on)
  • Ready for Staging (ready to be pushed to staging)
  • Failed on Staging (bugs that did not pass testing on staging)
  • Ready to Push live (Items that are ready to push live)
  • Completed (any tasks that have been been completed)

Note: a single client of ours can have more than one active project.

Software Testing Methods

  1. Acceptance Testing – This is ensuring that the application fits the client’s needs, and we work with the client to create these tests. This is testing the key performance indicators of the application.
  2. Unit/Component/Deployment Tests – These are standard software development/unit tests.
  3. Live User Testing – using active customers (NOT the client) to provide feedback and find issues and other “breaks” that would have never been thought of or planned.
  4. Application and Performance Testing – This includes learning the capacity of the application by testing the load times, application security, and so on.

Finally, we determine how we can automate as many of those tests as possible, to save time.