Part of the PurelyManage series.

Installing imapsync

imapsync is a Perl script. The version in most Linux package managers is years out of date and will fail on modern IMAP servers with current TLS requirements. Install it from the GitHub source instead.

First install the Perl dependencies via apt:

sudo apt-get install -y \
  libauthen-ntlm-perl libcgi-pm-perl libcrypt-openssl-rsa-perl \
  libdata-uniqid-perl libdigest-hmac-perl libdist-checkconflicts-perl \
  libfile-copy-recursive-perl libfile-tail-perl libio-compress-perl \
  libio-socket-inet6-perl libio-socket-ssl-perl libio-tee-perl \
  libjson-webtoken-perl liblockfile-simple-perl libmail-imapclient-perl \
  libmodule-scandeps-perl libnet-ssleay-perl libpar-packer-perl \
  libreadonly-perl libregexp-common-perl libsys-meminfo-perl \
  libterm-readkey-perl libtest-mockobject-perl libtest-pod-perl \
  libunicode-string-perl liburi-perl libwww-perl

Then download imapsync directly from GitHub and make it executable:

curl -o /usr/local/bin/imapsync \
  https://raw.githubusercontent.com/imapsync/imapsync/master/imapsync
chmod +x /usr/local/bin/imapsync

Verify the install:

imapsync --version

imapsync must be on the same machine as the PurelyManage backend. The backend spawns it as a child process and reads its stdout/stderr directly. A remote imapsync setup is not supported.

If you do not need IMAP migration, skip this step. All other features of the panel work without it.

How Jobs Run

When a migration job is created or requeued, it goes into an in-memory FIFO queue. The queue processor fills available slots up to MAX_CONCURRENT = 2:

const jobQueue: number[] = []
const runningJobs = new Map<number, ReturnType<typeof spawn>>()

function processQueue() {
  while (runningJobs.size < MAX_CONCURRENT && jobQueue.length > 0) {
    const nextId = jobQueue.shift()!
    runJob(nextId)
  }
}

runJob() decrypts the source and destination credentials from the database, builds the imapsync argument list, and spawns the process:

const args = [
  '--host1', job.src_host, '--port1', String(job.src_port),
  '--user1', job.source_email, '--password1', srcPass,
  '--host2', job.dst_host, '--port2', String(job.dst_port),
  '--user2', job.dest_email, '--password2', dstPass,
  '--nofoldersizes', '--noreleasecheck',
  ...(job.src_ssl ? ['--ssl1'] : ['--nossl1']),
  ...(job.dst_ssl ? ['--ssl2'] : ['--nossl2']),
  ...(job.dry_run ? ['--dry'] : []),
]

const child = spawn('imapsync', args, { detached: false })
runningJobs.set(jobId, child)

The --nofoldersizes and --noreleasecheck flags disable two operations that slow down every run without adding value: calculating folder sizes (slow on large mailboxes) and checking for a newer imapsync version (unnecessary in a managed environment).

detached: false means the child process is attached to the Node.js process. If the backend exits, imapsync exits too. This is intentional: a detached imapsync would continue running with no way to track its progress or update the job status in the database.

Streaming Logs to PostgreSQL

imapsync produces detailed output: connection status, folder-by-folder progress, message counts, transfer rates, errors. All of this is captured and written to the database in real time.

async function appendLog(jobId: number, chunk: string) {
  await sql`UPDATE migration_jobs SET log = log || ${chunk} WHERE id = ${jobId}`
}

child.stdout.on('data', (d: Buffer) => appendLog(jobId, d.toString()))
child.stderr.on('data', (d: Buffer) => appendLog(jobId, d.toString()))

Each chunk of output is appended to the log text column as it arrives. The frontend polls for new chunks using an offset:

GET /migration/jobs/:id/tail?offset=2048
→ { status: "running", chunk: "<new output>", newOffset: 3192 }

The frontend keeps track of how much of the log it has already displayed and only requests the new portion. At 1.5-second poll intervals this gives a near-live view of imapsync output without re-fetching the full log each time.

The full log is also available for download after the job completes, streamed as a plain text file.

Server Restart Recovery

The in-memory job state (the runningJobs map and jobQueue array) does not survive a server restart. If the backend process exits while a job is running, that job’s child process exits too, and the job row in the database is still marked running.

On every server startup, markInterruptedJobs() fixes this:

export async function markInterruptedJobs() {
  await sql`
    UPDATE migration_jobs SET status = 'interrupted', finished_at = NOW()
    WHERE status = 'running'
  `
}

Any job that was running when the server came back up is now interrupted. From the UI, interrupted jobs can be rerun the same way as failed jobs. The rerun creates a new job row and starts from scratch. imapsync handles duplicate messages gracefully by default, so re-running a partially completed migration is safe.

Jobs that were queued at the time of the restart are also lost from the in-memory queue. They remain in the database with status queued but nothing will start them. This is a known limitation: currently queued jobs need to be manually requeued after a restart by rerunning them from the UI. In practice this is rare since the systemd service restarts quickly.

Why Store Logs in PostgreSQL

Storing imapsync output in the database rather than on disk has a practical reason: the frontend can retrieve it over the same API used for everything else. There is no need for a separate log file endpoint, SSH access to read logs, or any special file serving configuration.

The tradeoff is that long migrations with verbose output can produce large text values in the log column. imapsync output for a large mailbox (tens of thousands of messages) can reach several megabytes. This is acceptable for a small-org tool but would need rethinking at scale.

The log download endpoint streams the value directly from PostgreSQL as text/plain with a Content-Disposition: attachment header, so it does not need to be held in memory:

reply.header('Content-Disposition', `attachment; filename="job-${id}-${name}.log"`)
return reply.send(job.log || '(no log output)')

The last post in this series covers deployment: getting the backend running as a systemd service, setting up nginx with TLS, and deploying the frontend to S3 with CloudFront.