Description

LongLLaMA is a large language model designed to handle long contexts, built on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method. The repository presents LongLLaMA as a research preview for context scaling and includes the model, related training code, and example notebooks.

The visible repository information also notes a smaller 3B base variant released under an Apache 2.0 license, along with inference code supporting longer contexts. The project includes code for instruction tuning and FoT continued pretraining.

How we innovate

LongLLaMA is distinctive for applying the Focused Transformer method to extend context handling, with the repository describing long-context capability up to 256k tokens in evaluation settings. It also offers a smaller 3B base variant under a permissive Apache 2.0 license and supporting code for continued pretraining and instruction tuning.

Use Case / Scenario

1. Long-context text understanding

Use LongLLaMA for tasks that need a model to work with extended input lengths, including research and evaluation on long-context behavior.

2. Model adaptation and fine-tuning

Use the repository’s instruction tuning and continued pretraining code as a starting point for adapting the model to custom language tasks.

3. Long-context inference workflows

Use the provided inference code and base model release to experiment with longer-context text processing in downstream applications.

Visit Website