How to distribute Mathematica packages as paclets?

The following answer is not complete, but does give one possible solution. There's a lot more to learn about the paclet manager, so please contribute another answer if you can, or correct this answer if you find any mistakes.

I originally posted this on Wolfram Community, following a nice tutorial by Emerson Willard on how to create paclets using Workbench. Most of the information is derived from studying GitLink.

To use Paclet Manager functions, it may be necessary to evaluate Needs["PacletManager`"] first.

Introduction

Packages can be bundled into .paclet files, which are easy to distribute and install.

.paclet files appear to be simply zip files that can contain a Mathematica package or other extensions to Mathematica, along with some metadata in a PacletInfo.m. The metadata makes it possible to manage installation, uninstallation and updating automatically.

I'm going to illustrate this using MaTeX. It is my smallest published package, so I used it for experimentation.

How to add the required metadata?

First make sure that your package is following the standard directory structure.

Then create a PacletInfo.m file in the package root with a minimum amount of metadata. Make sure that Name and Version are present. For MaTeX I could start e.g. with this:

Paclet[
    Name -> "MaTeX",
    Version -> "1.6.2",
    MathematicaVersion -> "10.0+",
    Description -> "Create LaTeX-typeset labels within Mathematica.",
    Creator -> "Szabolcs Horvát"
]

This is sufficient to make it possible to pack and install a paclet. But it is not sufficient for making it loadable with Needs. For that we need to add the "Kernel" extension:

Paclet[
    Name -> "MaTeX",
    Version -> "1.6.2",
    MathematicaVersion -> "10.0+",
    Description -> "Create LaTeX-typeset labels within Mathematica.",
    Creator -> "Szabolcs Horvát",
    Extensions -> 
        {
            {"Kernel", Root -> ".", Context -> "MaTeX`"}
        }
]

The two critical arguments to the `"Kernel"`` extension are:

  • Context sets the context of the package. Whatever you put here will be recognized by Needs and FindFile, but ideally it should also be compatible with the package name and the standard file name resolution.

  • Root sets the application root. FindFile seems to resolve the context to a path through this, but also following the standard rules.

Of course you can also add the "Documentation" extension to integrate with the Documentation Centre, but that is not required for the functionality I describe here.

Much more detailed information on PacletInfo files is here:

  • PacletInfo.m documentation project

How to bundle a package into a .paclet file?

Simple use the PackPaclet function on the application directory. It will use the information from PacletInfo.m. It is a good idea to remove any junk files and hidden files to avoid them from getting packed up.

Warning: Before doing this, make a copy of the application directory. Don't accidentally delete any files used by your version control system.

After making a copy of the package directory, in my case called MaTeX, I did this:

Make sure we're in the parent directory of the application directory:

In[2]:= FileNames[]
Out[2]= {".DS_Store", "MaTeX"}

Delete any junk files like ``.DS_Store` (which macOS likes to create): 

In[4]:= DeleteFile /@ FileNames[".*", "MaTeX", Infinity]

Create .paclet file:

In[5]:= PackPaclet["MaTeX"]
Out[5]= "/Users/szhorvat/Desktop/pacletbuild/MaTeX-1.6.2.paclet"

Install it permanently:

In[6]:= PacletInstall[%]
Out[6]= Paclet[MaTeX, 1.6.2, <>]

Multiple versions may be installed at the same time. Finds all installed versions using:

In[7]:= PacletFind["MaTeX"]
Out[7]= {Paclet[MaTeX, 1.6.2, <>]}

While this Paclet expression is formatted concisely, it contains all the metadata from PacletInfo.m, plus its installed location. You can see all this by applying InputForm to it.

FindFile (and therefore also Needs) will always resolve to the latest version:

In[8]:= FindFile["MaTeX`"]
Out[8]= "/Users/szhorvat/Library/Mathematica/Paclets/Repository/MaTeX-1.6.2/Kernel/init.m"

PacletFind will return the highest version first. To uninstall all but the highest version, we can use something like PacletUninstall /@ Rest@PacletFind["MaTeX"]. To uninstall all versions at once, 

PacletUninstall["MaTeX"]

How to work with paclets during development?

During development we don't want to pack the paclet and install it after every change. It is much more convenient to be able to load it directly from the development directory.

I can do this by adding the parent directory of the application directory (i.e. MaTeX in the above example) as a paclet directory. Since I keep the development version of MaTeX in ~/Repos/MaTeX-wb/MaTeX, I can simply use

PacletDirectoryAdd["~/Repos/MaTeX-wb"]

After this Needs["MaTeX`"] will load the dev version.

As a proof of concept and an experiment, I started distributing MaTeX in this format. You can use it as a reference in addition to GitLink.


Szabolcs answer shows how we can build a .paclet, but the PacletManager also contains the possibility to serve packages from an own site. As I realized after writing this, most of the information here can also be found in this Wolfram Community post, a link contained in Szabolcs PacletInfo.m documentation project.

Paclet Server Setup

The easiest setup is to use some webspace which serves static content. On that put a PacletSite.mz file into the root directory which contains information about which paclets and versions that site will serve. Add a directory named Paclets and put the paclets you did build as described by Szabolcs into that. The content of PacletSite.mz needs to be as follows:

pacletsite = PacletSite[Paclet[
  "Name" -> "PckgName", 
  "Version" -> "1.0.0", 
  "MathematicaVersion" -> "9.0+", 
  "Description" -> "A package to try the PacletSite functionality.", 
  "Creator" -> "your name",
  "Extensions" -> {{"Kernel", Root -> ".", Context ->"TryPacletSite`"}}
  ],
  ...

]

that is an expression with a Head PacletSite and as arguments a Sequence of Paclet expressions, which are basically the same as what is in a PacletInfo.m file, although I think you will need strings as labels here whereas the PacletInfo.m wants symbols, or at least some of the (java?) functionality that uses it like PackPaclet.

The PacletSite.mz can be generated from the above expression with:

Export["PacletSite.mz",{pacletsite},{"ZIP", {{"PacletSite.m", "Package"}}}]

Upload that and the paclet files to the server and test whether you can download them e.g. by visiting (of course you'll need to fill your own urls) "http://your.pacletsite.url/PacletSite.mz" and "http://your.pacletsite.url/Paclets/PckgName-1.0.0.paclet"

If that works you are set to experiment with the paclet manager.

Client Side Usage of PacletSite

this will show the currently configured paclet-sites, which should be just the wolfram research ones:

PacletSites[]

this will add your own paclet site url (for experimenting, I prefered to prepend it):

PacletSiteAdd["http://your.pacletsite.url","Description", Prepend -> True]

Note that PacletSiteAdd will add that url permanently, that means it will persist in the next session, you will need to use PacletSiteRemove to get rid of it.

The following will get the information about which paclets the given site serves, that is it will download and read the content of your PacletSite.mz:

PacletSiteUpdate["http://your.pacletsite.url"]

now it is possible to install a package from that site (optionally as shown using a specific version):

PacletInstall[{"PckgName", "1.0.0"}]

one installed, you can list all installed versions of a package:

PacletFind["PckgName"]

and of course load it:

Get["PckgName`"]

if you now put a newer version onto the server and also update the information in PacletSite.mz you can do:

PacletCheckUpdate["PckgName", "UpdateSites" -> True]

which will return a list of paclets for which the site now has newer versions than what you have installed. Using:

PacletUpdate["PckgName"]

will actually install the newest available version (if it is compatible with your Mathematica version). You should now see that in the list of installed versions and when loading you should get the new version:

PacletFind["PckgName"]

Get["PckgName`"]

to uninstall (all versions), you would do:

PacletUninstall["PckgName"]

check that all versions are gone:

PacletFind["PckgName"]

finally, to get rid of the added paclet site you would need to do:

PacletSiteRemove["http://your.pacletsite.url"]

I have no experience with how good this works in practice, I just did set this up and tried and it seems to work with version 9, 10 and 11. There seem to be some timeouts involved so you might get bad results if the server is too slow. If anyone makes own experiments I am very interested to hear how good that works for them. Of course all that functionality is undocumented with all the consequences that has. On the other hand it is the mechanism that WRI seems to use since at least version 9 to provide their own paclets so I would expect it should be fit for production...

Important: Security Considerations

As Szabolcs and Sjoerd C. de Vries have mentioned in their comments of course installing from an unkown web source has security issues. So when installing from external sources always be careful and act with a decent measure of mistrust.

The described setup does not actually add additional insecurities (you can already download and install Mathematica code from websources in other ways), but it of course makes it somewhat easier to get trapped into running malicous code.

The whole mechanism doesn't have any security measures and I don't see an easy way to provide one. To my understanding (I'm not a security expert), when you add a paclet site url as described you are trusting (at least):

  1. the package author to not provide malicious code,
  2. the web server which serves the paclets to actually serve the authors versions of these packages and
  3. your own computer and the DNS servers you are using that they correctly resolve the paclet-server address and not redirect to a malicous server.

What package managers for other languages or OS distributions usually do is to provide a certification mechanisms which make it much harder for malicious code to sneak in without the package provider realizing. AFAIK such a mechanism can prohibit attacks to 2. and 3.

Of course even with such a certification mechanism you still would trust the maintainer of the packages that their code won't do anything bad (I think there is no way to solve 1. technically)...


If you want to extend Albert Retey's answer to just use Wolfram tech you can set up your server in the cloud. I just set this up for myself as a proof-of-concept and it seems to work just fine.

Step 1: The Cloud

Get a free cloud account that you can put these into. Obviously restrictions will apply to the size of the packages you can distribute and whatnot, but looking at the pricing page you get .2 GB of space which, if you're mostly moving code base, not data, should suffice. If not see this answer to see how we can set this up using Google Drive too. The basic trick is to put up your paclet but provide an HTTPRedirect to a Google Drive download link.

Step 2: Deployment

Generate your application(s) that you want to move. I took some stuff I've developed, including the package that has the code that pushes to a cloud paclet, in an application I called AppSampler.

Configure it like you were gonna push to, say, GitHub, but now instead we're gonna push to the cloud.

(Note that if you just want to put your paclet up so that it's installable by PacletInstall you can just upload that--no need for this paclet site)

First we do our PacletSite.mz file:

co = CloudDeploy[None, "AppSampler/PacletSite.mz"];
CopyFile[ ".../AppSampler/PacletSite.mz", co];
SetPermissions[co, "Public"]

It's really just that last step that's in any way important, as it allows the paclet manager to access it. If I remember correctly from testing "Private" also works if you're on your own cloud account. Alternatively you can share with a group of people by setting up a PermissionsGroup.

Then we do the same for the paclets. In my case I just have the one, but you could do more:

co = CloudDeploy[None, "AppSampler/Paclets/AppSampler-0.0.paclet"];
CopyFile[ ".../AppSampler/Paclets/AppSampler-0.0.paclet", co];
SetPermissions[co, "Public"]

Step 3: Installation

Then after removing every trace of the paclet from out computers we do a PacletSiteAdd on the cloud repository, which, in this case lives at:

"http://www.wolframcloud.com/objects/user-affd7b1c-ecb6-4ccc-8cc4-4d107e2bf04a/_paclets/AppSampler/"

And then PacletInstall@"AppSampler" will pull in the paclet.

The great part of this is that it is a) free until WRI decides otherwise and (critically) b) possible to do entirely from Mathematica without having to link to any external resources.

There's something on the pricing page about a 30-Day limit on deployments, which is maybe applicable here -- does CloudDeploy count if it's really just to make the file exist for CopyFile? If it does that's a draw back, but probably non-fatal and at the very least this is still super convenient for temporary deployments. These paclets seem to survive in perpetuity.

Tags:

Packages

Development

Paclets

Recent Posts