GGUF and AWQ Model Files for Meta Llama 2s Llama 2 70B

Introduction

GGUF Format

GGUF is a novel format developed by the llamacpp team on August 21st, 2023. Its primary purpose is to calculate the cosine score between generated and reference texts using SentenceTransformers embeddings.

AWQ Format

AWQ, on the other hand, is an efficient, accurate, and rapid low-bit weight quantization method. It enables faster and more compact model deployment.

Model Merging and Exllama2 Quantization Guide

This repository offers a comprehensive guide for merging Llama2 70b models and implementing exllama2 quantization. It addresses common questions and provides step-by-step instructions for these processes.