Last Friday I updated the test results for my C++ refactoring test suite to include results from Visual Studio 2015 Update 1. Visual Studio 2015 is the first release of Visual Studio to support refactoring for native C++ projects without any additional add-ons. Prior to Update 1, Visual Studio requried a free add-on for Extract Function and Extract Method, but this has since been folded into Update 1 and made available to all Visual Studio users. Extract Function and Change Signature are disabled by default and can be enabled in Tools / Options / Text Editor / C/C++ / Refactoring.
As you can see from the summary results, Visual Studio did pretty well for a first release. I had previously attempted to run my test suite against Visual Studio 2015 when it came out and the tests all failed miserably. This seemed inconsistent with the quality of implementation I have come to expect from the Visual Studio team, so I emailed Microsoft to make sure that I was doing things properly. Gabriel Ha was kind enough to work with me on understanding what I was doing wrong. After a little bit of back and forth, we determined that my unexpected failures were due to my disabling of Intellisense in the IDE. I routinely disable Intellisense because it slows things down and I get very little value from it. The refactorings depend on Intellisense to get information about symbol context, definition, usages, etc. Once I enabled Intellisense and executed the tests again, the results were reasonable.
The test suite is by no means perfect and my discussions with Gabriel revealed some flaws in my test suite with respect to the one definition rule, which I have since fixed. Sometimes a failed test identifies a bug in a refactoring tool and sometimes a failed test reveals a bug in the test.
I usually end up making some improvements to the test suite every time I run the whole batch against a particular tool. This time, I added more executable unit tests around the test cases. This allows the test suite to identify a refactoring that failed because it changed the meaning of the code even though it still compiled. Usually a test fails by creating invalid code or by not being available for a particular test case. Another improvement is that I recorded before and after diffs for each test case to show the expected results of the test case. Different refactoring tools will do different things on formatting: are optional braces used, was the code indented with tabs or spaces, were end of lines written as LF or as CR LF, etc. So it is unlikely that the diffs will match perfectly between refactoring tools, but at least they are available for humans to consult in order to understand the intent of the test case.
So far, Visual Studio is the only tool to succesfully rename a label! Granted, you probably don’t have much use for labels in your code as you are unlikely to be making heavy use of
goto statements. I attribute Visual Studio’s success here to it’s leveraging of Intellisense. Compared to other tools in the test suite results, Visual Studio beat out Visual AssistX on Extract Function, Extract Method and Rename. Visual Studio also beat out CLion on Extract Method and Rename. ReSharper for C++ beat out Visual Studio on everything. Unlike those other tools, Visual Studio’s refactoring abilities are available for free in the Community Edition.
I welcome contributions to the test suite! The issues on github list areas where the test suite could be improved through new test cases or reported results for other tools. In particular, I am looking for someone who can contribute test results for Cevelop, Code::Blocks, Eclipse CDT, or QtCreator.