Researchers propose Zebra-Llama, a method to create efficient hybrid language models from pre-trained models, achieving Transformer-level accuracy with reduced training tokens and memory. Zebra-Llama outperforms existing models in accuracy and efficiency, using significantly fewer tokens and smaller memory.