Innovations in large language models are booming, with recent focus on using 4-bit activations for 1-bit LLMs. This strategy involves adding different activations to specific layers, like attention and feed-forward layers, to reduce computational costs while maintaining performance.