Introduction
Have you ever been typing in Chinese to your AI coding assistant, only to have it start replying in Korean? This puzzling behavior isn't random—it stems from how embeddings work under the hood. When code vocabulary mixes with natural language, the assistant's internal representation can drift, leading to unexpected language switches. In this guide, you'll learn how to investigate this phenomenon step by step, from setting up your environment to analyzing embedding spaces.
What You Need
- A computer with Python 3.8+ installed
- Access to an AI coding assistant API (e.g., OpenAI, Anthropic, or a local model)
- Python libraries:
numpy,scikit-learn,matplotlib,transformers(Hugging Face) - Sample prompts in Chinese and English, including code snippets
- Basic knowledge of embedding vectors and cosine similarity
- A text editor or Jupyter Notebook
Step-by-Step Guide
Step 1: Choose Your Testing Prompts
Select a set of prompts that mirror real-world usage. You'll want:
- Pure Chinese prompts (no code): e.g., "写一个函数来计算斐波那契数列" (Write a function to compute Fibonacci)
- Pure English prompts (no code): e.g., "Write a function to compute the Fibonacci sequence"
- Mixed prompts: Chinese with embedded code keywords like
def,return,if
Record the assistant's responses. Note any language shifts.
Step 2: Extract Embeddings from the Assistant
Most coding assistants allow you to access internal embeddings or you can use a separate embedding model. For example, using OpenAI's text-embedding-ada-002 or Hugging Face's sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embedding = model.encode("你的提示")Create embeddings for both your prompts and the assistant's responses.
Step 3: Analyze Embedding Similarity
Use cosine similarity to compare embeddings. The unexpected language switch often occurs when code vocabulary pulls the Chinese prompt closer to Korean-language embeddings in the model's space.
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
# Example: compare Chinese prompt with its response
prompt_emb = model.encode("写一个函数") # Chinese
response_emb = model.encode("함수를 작성하세요") # Korean
sim = cosine_similarity([prompt_emb], [response_emb])
print(sim)Key insight: High similarity between a Chinese+code prompt and a Korean response suggests the code vocabulary has bridged the language gap.
Step 4: Visualize the Embedding Space
Reduce dimensionality using PCA or t-SNE to plot embeddings. Color-code by language (Chinese, English, Korean). You'll often see a cluster where code-related terms mix languages.
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
# Assume you have a list of embeddings and labels
pca = PCA(n_components=2)
reduced = pca.fit_transform(all_embeddings)
for lang, color in [('Chinese', 'red'), ('English', 'blue'), ('Korean', 'green')]:
idx = [i for i, l in enumerate(labels) if l == lang]
plt.scatter(reduced[idx,0], reduced[idx,1], c=color, label=lang)
plt.legend()
plt.show()Step 5: Isolate Code Vocabulary Effect
Create a controlled test: Take a pure Chinese prompt and a pure English prompt about the same task. Then add identical code keywords (like for, while, import) to both. Compare the embeddings before and after adding code. If the Chinese+code embedding moves toward the Korean region more than the English+code does, you've found the culprit.
Step 6: Document and Repeat
Run your tests multiple times with different models (GPT-3.5, GPT-4, Claude, etc.). Note that each model's training data and tokenizer affect how code vocabulary reshapes language. Some models might switch to Japanese or other languages, not just Korean.
Tips & Best Practices
- Use consistent tokenization: Different tokenizers can break Chinese and Korean characters differently. Check token counts to ensure fair comparisons.
- Watch for false positives: A Korean response might sometimes be due to training data imbalance, not just code vocabulary. Cross-check with multiple prompts.
- Leverage existing research: Search for papers on multilingual embedding shifts in code assistants—this guide builds on that work.
- Adapt for prevention: Once you understand the pattern, you can ask the assistant to explicitly state its language, or use system prompts like "Always respond in the language of the user's input."
- Explore open-source models: If your assistant is a black box, test with alternatives like CodeBERT or StarCoder for deeper embedding access.
By following these steps, you'll not only decode why your assistant switched to Korean—you'll gain a practical method for analyzing any language drift in AI systems. Happy embedding!