My Gmail labels already described how I wanted mail grouped. I built mailsort to turn those labels into training data for a small classifier that runs on my own machine.

mailsort learns topics from a Google Takeout export, then checks incoming messages over IMAP and suggests Gmail labels. There is no hosted AI service classifying the message text. Gmail is still contacted to read messages and apply labels, but the model and classification run locally.

Teach it with labels you already use

The training data comes from an MBOX export made with Google Takeout. Existing Gmail labels become the categories, so I did not have to build a new labeling system just for the project.

The model turns message text into TF-IDF features and uses a calibrated linear classifier trained with hinge loss. I exported 27,316 messages; 3,107 had labels suitable for training across nine topic categories. Some labels identify people or accounts rather than topics, so I left those out. They overlap with many topics and can make the training examples ambiguous.

The first held-out evaluation reached 0.95 accuracy. That is a promising start, not proof that every category is ready. Results varied by label: the weakest F1 score was 0.64, and another category had 0.64 recall on just 14 held-out messages. Those are exactly the cases where a confidence threshold and a review label matter.

A cautious path to Gmail

mailsort supports Google app-password and OAuth authentication. After connecting over IMAP, it reads message bodies with BODY.PEEK[], which leaves unread messages unread. Gmail represents labels as folders over IMAP, so applying a topic label copies a message into the corresponding label folder. mailsort does not move or delete mail.

The default watch command is a dry run. It logs the label it would apply without changing Gmail. Messages below CONFIDENCE_THRESHOLD get the REVIEW_LABEL label instead of a topic prediction. Each decision is written as a JSON line to data/actions.log so I can inspect the results before enabling changes.

Set it up

  1. Export your mail. In Google Takeout, choose Mail, include the labels you want to use, and export as MBOX. Unzip the archive and put the file at data/takeout.mbox.

  2. Install the dependencies.

    python3 -m pip install -r requirements.txt

  3. Choose an authentication method. For an app password, enable two-step verification on your Google account and create an app password. For OAuth, enable the Gmail API in Google Cloud Console, create an OAuth client ID for a Desktop app, and save its JSON file as data/oauth_client.json. Configure the consent screen for your account. Apps left in Testing have refresh tokens that expire after seven days; publish the app to In production when appropriate.

  4. Configure .env. Copy .env.example to .env, set IMAP_USER, and set CATEGORIES to the exact Gmail label names to classify. With OAuth, leave IMAP_APP_PASSWORD empty.

  5. Train and inspect the model.

    python3 -m cli train
python3 -m cli evaluate

    train prints per-class metrics and a confusion matrix. evaluate reports in-sample metrics for the saved model, so use the held-out results from training to judge generalization. You can also smoke-test one message without changing anything:

    python3 -m cli test-one one.eml

  6. Start with a dry run. For OAuth, authorize once in the browser, then run watch:

    python3 -m cli auth
python3 -m cli watch

    Review data/actions.log. A dry run records processed messages in data/seen-dry.json, so the next run logs new mail instead of repeating the same decisions. When the predictions look trustworthy, enable label changes with:

    python3 -m cli watch --apply

The OAuth sign-in flow has been verified against Gmail in read-only mode, and the model has been trained on my archive. The next step is to configure the IMAP credentials, inspect a full dry run, and only then try --apply. I have not yet let mailsort label incoming mail automatically.

Run the tests

Install the development dependencies and run the test suite with:

python3 -m pip install -r requirements-dev.txt
python3 -m pytest

For now, mailsort is a trained local model with a cautious path toward live use. The dry-run log is the checkpoint between a good-looking metric and trusting it with an actual inbox.

Related Posts