🤝 Open to Collab

AbstractPhila PRO

AbstractPhil

19 6 27

https://civitai.com/user/AbstractPhila

AbstractEyes

AI & ML interests

datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.

Recent Activity

updated a bucket 23 minutes ago

AbstractPhil/alephllm-chat-storage

updated a Space about 1 hour ago

AbstractPhil/alephllm-chat

updated a model about 2 hours ago

AbstractPhil/mini-beatrix-1

View all activity

Organizations

Say hello to the Trigram ByteLLM - AlephLLM: Mini-Beatrix - in her huggingface space!
She is currently stepped at 24000 steps aka 7b tokens in the first couple datasets, so she's not very smart yet.
AbstractPhil/alephllm-chat

Be warned, whatever you say WILL be recorded in a public cache, WHEN the chat version works. For now she records nothing. The idea is to help debug the K/V cache, and I would rather the data accumulated be shared. If you wish to speak to her in private I will include a toggle, that way you'll see that nothing is recorded when you speak and you can still have a private chat with her. For now she's simply auto-completing, so have fun with her.

The AlephLLM prototype is currently in full training with SDPA attention.
AbstractPhil/alephllm-mini-beatrix-training

https://github.com/AbstractEyes/alephllm Here's the model code and training code for the prototype.
As the training progresses, the AlephLLM will become more coherent and communicative, the tensorboard will consist of a large series of useful and useless analysis, and each checkpoint recorded at around 2000 steps unless the train crashes or the system faults.

It will take about 9 hours for the first few datasets to converge, then I'll train a chat AMOE expert cluster to see if she wants to speak yet. Until then, she's learning.

Yes I know it's early, but there isn't much more I could think of to analyze the AlephLM directly currently. The only way train the AlephLLM, is to train the full AlephLLM prototype. The bigger training has to run, otherwise the analysis won't matter. As it progresses, the analysis and huge amount of tensorboard statistics will flood out. Everything is transparent through the process from start to finish, everything recorded.