Thursday 5 September 2013

Git Flow - Moving patches from one Commit into another Commit

This (longish) post will cover detailed git workflows and is part of the series of blog posts that show how we use the Git Flow workflow to manage TeamMentor's source code (you will also see practical applications of GitHub's powerful  of powerful features like Network Graphs and Pull Requests).

The key problem that we are going to solve, is the situation created by Michael Hidalgo’s TeamMentor fixes/commits/branches that were done against an commit (38bfcd54d8046372c0ace2409324ecc965761504) which was originally planed to be part of the next release, but we decided that the next 3.4 Release of TeamMentor will be based on the current 3.3.3 version (with is based on the earlier commit: b97a470ffa173d67a9c74373593eea03eb7a2da4).

The key reason is that he  38bfcd54d8046372c0ace2409324ecc965761504 commit (currently the parent of Michael’s fixes/branches) is not stable and is going now to be the basis of the 3.5_Release (this code contains a number of big changes which need more TLD and testing: native ASP.NET MVC routing, better Git support, native Markdown editor, depreciation of HTML WYSIWYG editor, and more)

In a nutshell, we need to re-apply Michael’s bug fixes to an earlier commit than the one used (i.e. backport those commits).


To start, here is what Michael’s branches look like at the moment (note that all have the 38bfcd54d8046372c0ace2409324ecc965761504  commit as parent):

image

Here is the commit (38bfcd54d8046372c0ace2409324ecc965761504) that we want to have as the parent, since this is the commit that is currently on the 3.3.3. release (and will be the basis for the 3.4 release of TeamMentor):

image

Basically, what we need to do is to 'just' backport the branches linked to 38bfcd54d8046372c0ace2409324ecc965761504 commit,  into the  b97a470ffa173d67a9c74373593eea03eb7a2da4 commit

Note: since this post was getting quite long, I moved some workflows into Appendixes (included below) so that the key actions/changes can be read in sequence.

Using the workflow described in the Appendix 1) Creating patches from Michael’s branches here are the patches to apply (i.e. these are all changes from the branches currently available in Michael’s dev repository):

image

After:

  • Fixing the master branch and creating a feature branch for the 3.5_Release (so that TeamMentor/Dev master branch is in sync with TeamMentor/Master master brach, and the 3.5 commits are not lost) 
    • ... see Appendix 2) Creating a 3.3.3 tag and branch in Dev repository 
    • ... see  Appendix 4) Creating a 3.5_Release Feature branch )
  • Applying the 6 patches that merged without conflict 
    • ... see Appendix 3) Applying patches
...we get the following TeamMentor/Dev 'not merged branches':

image

After the pull requests are made into a new 3.4_Release branch (see Appendix 5) Creating a 3.4_Release Feature branch for more details) we have 5 Issues/branches applied (and ready for QA):

image

Here is the graph view, with TeamMentor/Dev master (blue line below):

image

…. now being the parent of the Issue_142, Issue_51, Issue_400, Issue_475 and Issue_459 branches:

image

This concludes the (main part of) this post, which showed how to handle the scenario where  fixes (and branches) were applied to a commit whose release schedule was changed (and there was the need to back-port those changes into an earlier commit).

I think it is important to note that the workflow shown in here is a great proof of the power of Git (I can’t even image doing this in SVN).

In fact, in this case, we are paying the price for not being more formal in the use of Git Flow workflows, and for not being more strategic on where we applied simple fixes (like the ones shown here).

I.e. this should be easier next time.

That said, it took me orders-of-magnitude more time to write this blog post, than to actually make these changes/fixes :)



Appendix 1) Creating patches from Michael’s branches:

To create the patches, I grabbed a fresh clone of Michael’s dev repo (which is a fork of TeamMentor/Dev)

image

Then, on a git bash of this repository, I created a new branch that pointed to the current
38bfcd54d8046372c0ace2409324ecc965761504 commit, using the commands: $ git checkout 38bfcd54d8046372c0ace2409324ecc965761504 and $ git checkout -b Patch_Parent

image

The reason I picked the 38bfcd54d8046372c0ace2409324ecc965761504 commit is because this is the commit that all Michael’s current branches are based on:

image

Using the $ git branch -a command, we can see that this local repository/clone already contains the branches we need:

image

Let’s start with a simple one, for example the changes on Issue 534:

image

whose changes are on branch Issue_534

image

In order to create the patch, I created a local tracking branch using the command $ git checkout -b Issue_534 remotes/origin/Issue_534

image

I then created a patch using $ git format-patch Patch_Parent

image

…which created the file 0001-Fixing-Issue_534.patch:

image

… containing these changes:

image 

…the these ones:

image

Note: the reason the patch is about 1Mb is because Michael (on this branch) also committed a bunch of *.dlls which should not be there.

One more little thing, since we are going to create a number of these patch files, it is better to put them on a dedicated folder. This can can be done using the command: $ git format-patch Patch_Parent -o ../_3.4_Patches

image

… with the 'patch file' now being placed on the folder:

image

Here is the same process for Issue_565:

image

… with the patch created in:

image

We can also create the patches without creating a tracking branch. For example here is how to create a patch for the code at the Issue_51 branch:

image

Note that the 0001-Fixing-Issue-51.patch file is much smaller (3k) then the others

image

This is caused by this patch only containing text diffs (and no binaries), which is how all patches should be:

image

Finally here are all the patch files created (containing all commits made by Michael's branches):

image


Appendix 2) Creating a 3.3.3 tag and branch in the TeamMentor/Dev repository

In order to be able to apply the changes into the TeamMentor/Master master branch, I created a branch in the current TeamMentor/Dev that points to the last common commit between TeamMentor/Master and TeamMentor/Dev (this way the commits can be pushed into TeamMentor/Master master branch, and eventually pulled into the TeamMentor/Dev 3.5_Release branch)

Since b97a470ffa173d67a9c74373593eea03eb7a2da4  is the last commit in TeamMentor/Master that also exists in TeamMentor/Dev, we are we are going to use as the parent for the patches/branches to apply:

image

To do so, I started by opening up my local dev repo (currently in sync with the latest commit to Dev) , and executed $ git checkout b97a470ffa173d67a9c74373593eea03eb7a2da4

image

I then created a tracking branch (called 3.3.3_Release) and added a tag (called v3.3.3), using the commands: $ git checkout -b 3.3.3_Release and $ git tag -a v3.3.3  -m '3.3.3 Release'

image

I then pushed the 3.3.3_Release branch and v3.3.3 tag into the TeamMentor/Dev repository, using the commands: $ git push dev 3.3.3_Release:3.3.3_Release and $ git push dev v3.3.3

image

Following these commands (and without the pushes that will happen next) we can see the 3.3.3_Release tag in TeamMentor/Dev network graph
image


Appendix 3) Applying patches

We are now going to apply the patches files (previously created), into the 3.3.3_Release branch of the current local clone of TeamMentor/Dev

image

Starting with the 0001-Fixing-Issue_142.patch which is a simple change:

image

To get a preview of what will change when we apply a patch, we can use the command: $ git apply --stat ../_3.4_Patches/0001-Fixing-Issue_142.patch

image

To see if we are going to have any errors when applying a patch, we can use the  command: $ git apply --check ../_3.4_Patches/0001-Fixing-Issue_142.patch

image

In this case, the fact that we saw no messages on the –-check command (shown above), means that we can merge this patch file ok:

image

... in this case the change was applied on top of our current branch code (with no commit added)

image

But that has the problem that there was no commit made (just the files changed on disk).

Since we want to preserve the original commit we, will need to can use another command.

First lets reset the current change:

image

... and before we apply the 0001-Fixing-Issue_142.patch, lets create the Issue_142 branch, using the command git checkout –b Issue_142

image

Now lets apply the patch this using the command: $ git am --signoff < ../_3.4_Patches/0001-Fixing-Issue_142.patch

image

...which will add a commit containing the original commit message and author:

image

Next we push this branch into TeamMentor/Dev

image

And confirm that the Issue_142 changes are in the correct location (i.e with the b97a470ffa173d67a9c74373593eea03eb7a2da4  commit as its parent):

image

Note how the light blue line is connected from the b97a470ffa173d67a9c74373593eea03eb7a2da4  commit (see above) into the newly pushed 5319e3028da01c64d09409b833c4f33bc49b7208 commit (see below)

image

... which is the current head of the Issue_142 branch

image

The next image shows how we can use GitHub's UI to create/view the pull request for this branch:


image

Note how in the screenshot above the Issue_142 branch is 129x commit behind master.

That is caused by the fact that master is currently at the commit 16354b3ec1757f56f0ee1594de3c72bb506f6537 and it should be at the commit b97a470ffa173d67a9c74373593eea03eb7a2da4

See Appendix 5) Creating a 3.4_Release Feature branch and merging branches for how that was fixed.

After mapping the current master commit into a new the 3.5_Release branch and doing a force reset to the master branch, we get the Issue_142 branch correctly set-up with 1x commits ahead and 0x commits behind the master branch:

image

With TeamMentor/Dev master branch in the correct location, lets apply more patches into it:

For example Issue 51, using the commands:
$ git apply --check ../_3.4_Patches/0001-Fixing-Issue-51.patch  (check if patch can be applied)
$ git checkout -b Issue_51                                                              (create patch branch)
$ git am --signoff < ../_3.4_Patches/0001-Fixing-Issue-51.patch  (apply patch and preserve original commit)
$ git push dev Issue_51:Issue_51                                                  (push branch into GitHub)
image

This makes the Issue_51 branch to also be 1x ahead and 0x behind commits of the master branch:

image

With this workflow in place, I quickly did the same workflow for the branches: Issue_384 , Issue_400 , Issue_475 and Issue_459

At the moment we have these branches to merge (Appendix 5) Creating a 3.4_Release Feature branch and merging branches will show them in action):

image

Note that there were numerous patches (534, 565, 193, 285, 461, 517, 504, 527,462 and 445) that didn’t merge correctly.

For example this is what happened for the 0001-Fixing-Issue-565.patch when executing the command $ git apply --check ../_3.4_Patches/0001-Fixing-Issue-565.patch

image

These will need to be handled separately (which is a topic for another blog post, since this one is already getting a bit long :) )


Appendix 4) Creating a 3.5_Release Feature branch

In order to make the current TeamMentor/Dev match the TeamMentor/Master in terms of the master branch, we need to move the current master of TeamMentor/Dev into a feature branch called 3.5_Release (in a way we were using the master of TeamMentor/Dev as a ‘feature branch’ which was ok if that code was going to become the 3.4 release (which now it isn’t).

First step is to move into the current master using the command $ git checkout master

image

Then we create the 3.5_Release feature branch using the command $ git checkout –b 3.5_Release

image

Next we push this branch into TeamMentor/Dev

image

At this moment, in the GitHub repo, TeamMentor/Dev's master and 3.5_Release point to the same commit (16354b3ec1757f56f0ee1594de3c72bb506f6537):

image

Now comes the sledgehammer :)

We’re going to (first locally) do a hard reset into the b97a470ffa173d67a9c74373593eea03eb7a2da4 commit, using the command $ git reset --hard b97a470ffa173d67a9c74373593eea03eb7a2da4 (remember that this commit is the common one between TeamMentor/Master and TeamMentor/Dev)

image

After this hard reset, the TeamMentor/Dev master is aligned with the 3.3.3_Release branch and v3.3.3 tag (previously created)

image

We can also double check this, by using the command $ git gui

image

… followed by the Visualize all Branch History menu option:

image

…and see that the Issue_142 branch is now a child of the current TeamMentor/Dev master (which is in sync with the TeamMentor/Master master)

image

Finally we are ready to apply the sledgehammer to the repository hosted at GitHub, by forcing a push using the command $ git push –f dev master:master

image

Which makes the TeamMentor/Master look like this:

image

... with the Issue_142 branch having the master/3.3.3_Release branch as parent (see rouge/brown line)

image

... and the 3.5_Release branch containing the commits that ware previously in the master branch (see yellow line)

image

Finally a look at the current branches in TeamMentor/Dev shows that the Issue_142 is correctly 1x commit ahead and 0x behind the master branch (which means that it is ready for a pull request)


image

Appendix 5) Creating a 3.4_Release Feature branch and merging branches

At this point we have these branches ready to commit (via a pull request)

image

Instead of merging them into the TeamMentor/Dev master branch, we are going to create a TeamMentor/Dev 3.4_Release branch using the command $ git checkout -b 3.4_Release and push it to TeamMentor/Dev using the command $ git push dev 3.4_Release:3.4_Release

image

The reason for this branch is so that TeamMentor/Dev master branch is aligned with TeamMentor/Master master branch (which is the current official release), and only QA'd changes are pushed into TeamMentor/Master (first into 3.4_Release branch, and eventually into the official TeamMentor/Master master branch (note that we will most likely rename the TeamMentor/Master repo into TeamMentor/Release)

Next step is to create a pull request from the current Issue_XYZ branches into the 3.4_Release branch.

Let’s start with Issue_459, by clicking on its Compare button:

image

On the next page, click on Edit:

image

... to change the base branch (into 3.4_Release):

image

Then click on the Click to create a pull request for this comparison link

image

… click on the Send the Pull Request button:

image

… click on the Merge pull request button

image

… and the Confirm Merge button:
image

We could now delete the branch (but I’m not going to do that at this stage, since first I want to see these merged branches in a GitHub Network Graph):

image

Back into the Branches not merged into master list, although the Issue_459 branch is still 1x ahead of master, we now have the 3.4_Release branch with 2x commits ahead:

image

The two commits of the 3.4_Release branch are one from the Issue_459 branch and one from the pull request merge (note above how we could now do a Pull request from this 3.4_Release branch into the master branch):

image

After doing the same workflow for Issue_475 branch:

image

... the 3_4_Release branch is 4 commits ahead:

image

And after doing the same workflow for the Issue_142, Issue_51 and Issue_400 branches/issues, the 3_4_Release is 10 commits ahead (with 5 Issues_Xyz applied):

image

The TeamMentor/Dev graph also shows this workflow in action (note that If I had deleted branches after the pull request, we wouldn’t see the tags in this network graph)

image

One important note is that the Issue_384 didn't merge automatically with the 3.4_Release, which means that there is a conflict between one of the changes made by the applied branches and this code (i.e. Michael will need to fix this and resubmit the patch)

image


Wrapping up: Feedback and better git commands:

If you made it this far to the end, it would be great to have some feedback on this git workflow (and solution).

And if you know of better ways to do solve probs like this one, please ping us with your ideas, since there is still far too much Git functionality that I/we are not aware of.