Moving Large Program Data to Another Drive with Links

I regularly run a program (on Windows) that downloads and indexes quite a bit of data. While this makes the application run quite smoothly, it does take up a lot of disk space. Recently, a change modified the save location for this data from my spacious D:\ to the rather more cramped conditions of C:\Users\foggy\AppData\Local. While this may be a more appropriate location, it contributed to filling up my C:\…and still asked for just a bit more space… So, now I had a dilemma, with multiple programs competing with each other over the limited C:\. Perhaps a bigger drive might be in order? Or, perhaps, there’s a better way?

Introducing linking. Linking provides a means to connect files and directories across the filesystem. The one most familiar to Windows users is the ‘Shortcut’, accessible through the right-click context menu. A shortcut simplifies navigating the UI (i.e., Windows Explorer). For example, within a work environment, a person might be involved in several projects, each with a long or complex path on a shared drive. To more easily locate the particular folders, the user can right-click the target directory, and choose ‘Create Shortcut’, moving the shortcut to any easily accessible location (e.g., Desktop). This type of linking is primarily UI-based (e.g., from a shell, you can’t cd target - Shortcut.lnk.

Making Links with mklink

These UI-based shortcuts won’t provide a useful point for my application. Instead, we’ll want a ‘deeper’ association, not just one file which links to another. This is where the mklink command will help us out. mklink is a Command Prompt command and may require administrator access, so let’s open it as admin by clicking the Windows Button > Type ‘cmd’ > Choose ‘Run as Administrator’.

Running mklink /? we can see the three options: /D for symbolic (or ‘soft’) link, /H for hard link, and /J for a ‘Directory Junction’. Let’s explore each of these in turn.

It’s worth noting as well, that a short and (perhaps) more useful summary is available in the documentation here: https://ss64.com/nt/mklink.html.

Directory Symbolic Link (‘Soft Link’)

The first option is also the default. So if you forget to include an option, you’ll end up with the soft link. We can create a symbolic link by running mklink /D symbolic_link target. The target can be a file or directory (and a directory/file cannot have the symbolic link’s name. The symbolic_link and target can also be across drives/file systems.

We can test this by navigating (in a terminal) to D:\test\temp\symbolic_link, create a file, navigate to ..\target and see the same file. How does this work? The ‘soft link’ is pointing to the name of the target, so if target is renamed or deleted, the link is broken. If it’s a directory, we can cd to symbolic_link, but a run of ls will result in an error.

Fixing (or re-creating) target will make the link work again. The name ‘soft’ or ‘symbolic’ emphasizes the superficial nature of the connection: the link is just looking for a name. This is not what we want. I want both target and link to point to the same inode on the filesystem, not just the name. For this we’ll need hard links.

Hard Link: Deeper Connections

Rather than pointing to a name or path, hard links create a link to the underlying data. With a hard link, the target can be renamed or even deleted, and the link will remain. Unlike with symbolic links, hard links are limited to files only. Attempting to link a directory will result in the cryptic Access is denied error.

So, let’s create a target.txt and link with mklink /h hard_link.txt target.txt.

Open these in a text editor that detects changes (e.g., not Notepad; use Sublime Text, VS Code, or JB Fleet). Typing something (and saving) in either will ‘update’ the other since these are both pointers to the same underlying file. What happens if we delete the original target.txt? hard_link.txt still contains the same text — it’s pointing to the same underlying data (inode) which will not be removed since it still has a pointer.

Unfortunately, this solution will also not work. I need to link directories.

Directory Junction

The directory junction is apparently a type of symbolic link. The differences between this and the regular symbolic links are a bit opaque to me (see a very nice discussion here: https://superuser.com/questions/343074/).

It does, however, have several advantages (in my case) including reduced security requirements (e.g., no admin perms required to create) and might process a bit quicker.

To do this, we just give the /J argument.

Implementing the Solution

As I mentioned above, my directory already contains a lot of data. Rather than deleting and restarting, I followed these steps:

• Move the directory C:\Users\foggy\AppData\Local\AppName to D:\AppName.
  • There is now no C:\Users\foggy\AppData\Local\AppName directory
• Once the copying completes, run mklink with the /J directory junction argument
  • mklink /J C:\Users\foggy\AppData\Local\AppName D:\AppName
• Test by running: cd C:\Users\foggy\AppData\Local\AppName and you’ll end up with the contents of D:\AppName.

And, best of all the application has quit it’s complaining, not realizing that it’s actually laboring away on a different planet.