Live CD and USB operating systems are a great tool for working on computers with a slow or broken OS, they let you bring all your tools with you and don't need to make any changes to the host PC in order to operate. There are a number of Linux LiveCD distributions that are built for general use or specific tasks like file recover or drive partitioning but most of them lack "persistence." When you boot up a live CD/USB it is always the first boot, changes you made the last time you ran it do not carry over to the next.
Although there being a few live distributions that do have persistence, Debian, which I am most familiar with, is not one of them. After some moderate Googling I found a tutorial here: Creating a Debian Live Iso-Hybrid USB key with persistence which shows how to take the non-persistent Debian Live ISO and give it the ability to save any changes you make.
The process involves making 2 partitions on a USB stick, one to hold the operating system and one to hold any changes you make when you run the OS. The Debian Live ISO is extracted into the first partition and a few changes are made. Note in step 4 there is no longer a /syslinux/live.cfg and the word "persistence" was added to live boot option in /syslinux/menu.cfg instead. After booting the USB stick for the first time I also found it necessary to change the repository URLs in /etc/apt/sources.list to my local mirror in order to install new software.
With those two deviations from the original tutorial I was able to get a full system that can have software added and files stored and will run on nearly any PC.