Is it possible to improve Mathematica's auto-completion feature?

Preface

With Mathematica version 9.0.1 the following answer not valid anymore because the underlying protocol between front end and kernel was changed. Fortunately, we started to implement an open-source Mathematica plugin for IntelliJIDEA which has a full support for camel-hump completion.

Please see this post for more information

Camel-humps auto-completion for Mathematica

The answer to my question is yes, we can expand the auto-completion capabilities of Mathematica. For those who want to know, how exactly this can be done, I suggest reading the Reverse engineering section below. There I explained how you can investigate in the default completion in Mathematica and how you can modify its behavior.

For all those who cannot wait to test it: A simple Get of my package from my GitHub repository turns the camel-humps completion on:

Get["http://goo.gl/wtVze"]

In Mathematica 9 you should now see a suggestion box when you type e.g. PP. If you want to turn it off again can simply restart the kernel or you use FE`RecoverDefaultCompletion[]. If you want to install it permanently you can for instance copy the contents of the package to your init.m.

Usage

To be really fast with this kind of help you have to try avoiding arrow-keys for going down the suggestion list. Instead, you should specify your wanted function further and further by leaving out the small letters between capital camel-hump letters. A very good example is Hypergeometric0F1. Instead of typing Hy and going down the list very far, you leave out 12 letters and type Hy0. In many, many cases this leads to only one unique choice of an expansion and you only have to press Enter (or Tab) to accept it.

If there are still several choices in the end, then you can use your arrow keys. One example here is, when you typed (heroically) LLLP to shorten ListLogLinearPlot and see that there is a ListLogLogPlot with the same abbreviation. Unfortunately, at this point even typing LLLPlot does not help to make it clear and you have to use the arrow key to choose the right function.

Let me show you, what works and how it works. Completion of usual function names works now both ways:

1. You get valid expansions if you type a prefix of the function like Integ for IntegerDigits.
2. Additionally, you can type IntDig, InDi or just ID to get IntegerDigits too. Important is that you take care of the capital letters in the names.

Support for completion of Options was added too.

Additionally, camel-humps even work after context specifications. Note, that the context-names themselves cannot be camel-humps expanded because in most cases like Developer, Experimental, Internal, etc. this would be useless anyway.

How it was reverse-engineered

Let me explain in detail how you can investigate into how the auto-completion works. What I used is the JLink Link Snooper. With this you don't start the Mathematica kernel as usual. Instead a special JLink program listens and prints the communication between front-end and kernel.

The installation is straight forward: You go to Evaluation->Kernel Configuration Options and create a new kernel exactly as described in the documentation to LinkSnooper I linked above. Then you set either Default Kernel or the Notebook Kernel (in the Evaluation menu) to the LinkSnooper and restart the kernel. A java window pops up showing you all traffic. When the message-war of the initialisation is over you can clear the window and then type for instance Pl and you see that at this very moment the front-end sends a packet to the kernel. The LinkSnooper tells me

System`EvaluatePacket[FrontEnd`SynchronousDynamicEvaluate[TimeConstrained[
   FE`CAFC["Pl", False]
    , 6.0], 0, 9.0, 0.2563962936401367]]

The important line is of course the second one. Here you see, that the function FE`CAFC is called. Let's have a look a the implementation (I killed all context prefixes in the output):

??FE`CAFC

CAFC[nameString_,ignoreCase_:False]/;$Notebooks:=
  MathLink`CallFrontEnd[FrontEnd`CompletionsListPacket[
   nameString,
   Names[nameString<>"*",IgnoreCase->ignoreCase],
   Contexts["*"<>(nameString<>"*")]
  ],NoResult]

The kernel sends a CompletionListPacket to the front-end which has 3 parameters. As far as I could find out they mean the following:

1. nameString is not only a repetition of the input-string! This is used as filter. So when Pl is expanded and the list of suggestions (the second parameter) would contain the word BarbieAndKen the front-end won't show this in the choice-box, because Barbie.. does not match Pl.... This is important for our camel hump expansion, because we try to expand something like LLP to ListLinePlot. The solution here is to take only the first letter of the input-string which will hopefully always match. So if I try to expand LLP and I give ListLinePlot in my suggestions list, then the L... will match and ListLinePlot is displayed in the suggestion box.

2. The second parameter is simply a list of suggestions which are displayed if they match the first parameter. Currently, only all matching names are displayed. We will extend this, so that we calculate possible function-names from camel hump shorties.

3. This is the list of possible matching context names.

How options-matching works

Unfortunately, the matching of options seems to works differently. Here, Mathematica uses the syntax information of the function to decide whether you are on a possible option position. Here are two equivalent calls:

Plot[x, {x, 0, 1}]
Plot[Sequence[x, {x, 0, 1}]]

If you now go behind the closing curly braces of each example, make a comma and type Pl you shouldn't be too surprised that the first list contains all options, where the second choice-box contains usual function. If you check the output of the LoopSnooper, then you see that only in the first case the function FE`OC["Plot", "Pl"] is called. This is the option-completion function and it looks like

OC[nameString_,patternString_]/;$Notebooks:=
  MathLink`CallFrontEnd[FrontEnd`OptionCompletionsListPacket[
    nameString,
    patternString,
    (ToString[#1]&)/@(InputForm[#1]&)/@
      Options[ToExpression[nameString]]],
  NoResult]

See, that here a OptionCompletionsListPacket is send to the front-end. Therefore, the front-end handles options-expansion differently. For the moment I will not touch this, because I haven't tested the stuff, but I will. What you should see here is, that our camel hump expansion will currently only work on function names and not, when you are inside a function at an option-position.

Implementation

Here are the important details of the following lines. First, I will import the file with the function informations of all Mathematica functions. This contains function names, call patterns and template constructs, but I will only use the function names. The difference to a call to Names is, that this contains the function names only and not some options. For the current purpose this is maybe better.

The next one-liner is maybe the most important function:

nameParts[in_String] := StringCases[in, _?(UpperCaseQ[#] || DigitQ[#] &) ~~ ___?LowerCaseQ];

This has the purpose to split a Mathematica-style function name into its parts. When a capital letter is seen, a new part starts

nameParts["ListPointPlot3D"]

(* {"List", "Point", "Plot", "3", "D"} *)

What you could do is, you take from every part as many lower case letters as you want from the back and put it back together and you have a valid, expandable camel hump short-cut. Important is, that such a word-part starts always with a capital letter and is followed only by lower-case letters. Numbers are like capital letters. This is then used in the getMatch function. There, we take some input and calculate all possible function expansions by using the name-parts to create a regular expression.

Then we implant this into the FE`CAFC function and after executing this, the advanced expansion should work. So typing FiAM for instance gives

The complete implementation can be found on my GitHub site

Known issues and limitations and bug-report

To report bugs or unusual behavior please use the issues section here.

• The front-end does not call on every keystroke the auto-completion function. You can watch this on the LinkSnooper. This means for instance, that you won't get a choice box when you type FAM, although FindArgMin matches. Our function is simply not called. What you can do is type another i or press Cmd+k.

• On Linux there seems to be a bug regarding the above point: When you type Pl and close the appearing suggestion-box with Esc it is not possible to use Ctrl+k to reopen this box. Using arrow-keys to move around and going then to the same place and it works. 

• jensbob reported that a message was generated about Throwing False when the code is put into init.m. This issue is fixed and was not related to the autocompletion code itself.

• sebhofer asked in the comments: Would it in principle be possible to get completion similar to most shells where hitting tab completes to the longest common sequence and not to the first match? Answer: In principal yes, but there's a very big but attached to this. In version 8 where you have to hit Ctrl+K for expansion the suggestion-box shows the entries exactly in the same order as I supply it (at least in Linux!). Here, the thing you would have to implement is the calculation of the longest common sequence and then you prepend it to the list of suggestions you return. Then this is always on top of of list and you can simply expand it always. In version 9 the list of suggestions is sorted by the FE (front end) and I don't know how to influence the ordering. This all happens inside the FE where you have not really a chance to re-engineer what happens. The only possibility I see here is to supply the longest common sequence as only suggestion when it exists. One problem in both versions is that the FE closes the suggestion and don't asks the kernel again whether there exist more possible expansions because it believes you already found the function you like to have. You can test this: type Plo and you see the Plot is LCS here. Accepting this closes the window and you would have to type another letter or Ctrl+K to reopen it. I don't think this is a good idea.

• 18/01/2013 I forgot to include $ as separator and therefore variables like $KernelID can not be shortened by e.g. $KID. Furthermore, I think about adding symbols in the Global` which don't start with a capital letter. In this way it's possible to expand user-defined symbols too.

In the meantime, I was playing around. This is just to add a hotkey (Alt+k in windows) to replace what you have written so far with the partial symbol found. I don't know if it is a useful thing if we don't add it a way to handle multiple findings.

Put this in a "init.m" file, inside FileNameJoin[{$UserBaseDirectory, "Applications", "AutocompleteBonus"}]

BeginPackage["AutocompleteBonus`"]

CamelhumpExpand[] := With[{nb = InputNotebook[]}, 
  FrontEndTokenExecute["ExpandSelection"]; NotebookWrite[nb, 
    NotebookRead[nb] /. s_String :> With[{
        spatt=StartOfString~~StringReplace[s, b:Except[StartOfString]~~
              c_?UpperCaseQ :> b~~___?LowerCaseQ~~c]},
      With[{new = Names[spatt~~"*"]}, 
       StringCases[First[new], spatt, 1] /; new =!= {}]]]]

EndPackage[]

Now, add (backup first), in MenuSetup.tr which is in some subfolder in FileNameJoin[{$InstallationDirectory, "SystemFiles", "FrontEnd", "TextResources"}], a line such as the following one

    MenuItem["Cameltoe E&xpand", FrontEnd`KernelExecute[Needs["AutocompleteBonus`"];AutocompleteBonus`CamelhumpExpand[]], MenuKey["k", Modifiers->{"Command"}],MenuEvaluator -> "System"],

right after (for example), the line that starts with MenuItem["Complete Se&lection"

Restart MMA, and test.

EDIT

@halirutan, failure, and gotta go. i was trying somethign along this lines, but as you suggested, dynamic

getCandidates[] := With[{nb = InputNotebook[]}, 
    FrontEndTokenExecute["ExpandSelection"]; 
      NotebookRead[nb] /. s_String :> With[{

      spatt = StartOfString ~~ 
        StringReplace[s, b : Except[StartOfString] ~~
                         (c_)?UpperCaseQ :> b ~~ ___?LowerCaseQ ~~ c]},
           With[{new = Names[spatt ~~ "*"]}, 
             new ]]]

suggestStuff[] := 
 MathLink`CallFrontEnd[FrontEnd`AttachCell[First@SelectedCells[],
   Cell[
    BoxData@ToBoxes@
      Row@
       Riffle[Button[#, NotebookWrite[First@SelectedCells[], #]] & /@ 
         getCandidates[], Spacer[20]]], "Inline", 
   "ClosingActions" -> {"SelectionDeparture", 
     "ParentChanged", "EvaluatorQuit"}]]

Now you write in a cell something like "LP", in a different cell evaluate Pause[2];suggestStuff[], and you have 2 seconds to put your selection back after "LP"

I'm using Mathematica 12.1 and I think the following feature was introduced in version 12.

You can add and execute the following code in your notebook:

(* the default value is Automatic *)
CurrentValue[EvaluationNotebook[], {CodeAssistOptions, "MatchingAlgorithm"}] = "Fuzzy";

Source: YouTube Wolfram Technology Conference 2019: Stephen Wolfram's Keynote Address at 2h 23min.

With this option the Mathematica code completion becomes much better which can be seen below