Parcourir la source

add clarity on EOS token

Suraj Subramanian il y a 7 mois
Parent
commit
71cbaa29e9
1 fichiers modifiés avec 2 ajouts et 1 suppressions
  1. 2 1
      README.md

+ 2 - 1
README.md

@@ -7,8 +7,9 @@ The 'llama-recipes' repository is a companion to the [Llama 2 model](https://git
 > | Token | Description |
 > |---|---|
 > `<\|begin_of_text\|>` | This is equivalent to the BOS token. |
-> `<\|eot_id\|>` | This signifies the end of the message in a turn. This is equivalent to the EOS token. |
+> `<\|eot_id\|>` | This signifies the end of the message in a turn. |
 > `<\|start_header_id\|>{role}<\|end_header_id\|>` | These tokens enclose the role for a particular message. The possible roles can be: system, user, assistant. |
+> `<\|end_of_text\|>` | This is equivalent to the EOS token. On generating this token, Llama 3 will cease to generate more tokens |
 > 
 > A multiturn-conversation with Llama 3 follows this prompt template:
 > ```