Uploading Files

On the upload screen, drag and drop your .txt files or use the file picker. Each file you upload is treated as a separate document in the topic model. The more files you include, the more stable and interpretable your results — but there are practical limits to ensure performance and readability.

How File Uploads Work

Each file represents one document. The model learns themes across the corpus by analysing word patterns across these documents. If your files are too short (under ~300 tokens), topics may feel weak or generic. If they are too long or unevenly distributed, dominant documents may skew topic prevalence.

Use consistent file formatting and avoid uploading empty, duplicate, or placeholder files. File names are displayed in the Report tab, so name them meaningfully to track interpretations. Filenames in any language — Bengali, Hindi, Tamil — are fully supported and will display correctly in all charts.

What Not to Upload

• PDFs, Word docs, or ZIP files — only plain .txt files are accepted
• Files with non-UTF-8 encodings — these may break preprocessing or display gibberish
• System files (like .DS_Store) — these will be ignored or discarded

Tips for Best Results

• Use UTF-8 encoding (check in your text editor or use Save As → Encoding)
• Avoid OCR scans or pasted content from PDFs unless cleaned (see Corpus Preparation)
• If a document contains less than 100 tokens after filtering, it may be skipped automatically
• Use short, descriptive filenames to help interpretation later — e.g. tagore_ghare_baire.txt, premchand_godan_ch1.txt

Stopwords

Stopwords are high-frequency words — such as pronouns, particles, or auxiliary verbs — that tend to dilute topic modelling results. Removing them sharpens topic coherence by reducing statistical noise.

How anvay Handles Stopwords

anvay offers two mechanisms for stopword removal:

1. Use the built-in NLTK stopword list
   Enable this by selecting Yes on the Remove Stopwords field. NLTK provides stopword lists for BN Bengali and EN English. For HI Hindi and TA Tamil, this option has no effect — NLTK does not include these languages.
2. Upload a custom stopword file
   In Advanced Options, upload a plain-text .txt file with one word per line. This replaces the built-in list entirely for that run.
   
   anvay provides ready-made stopword lists for Bengali, Hindi, and Tamil. Use the Download built-in stopwords list link next to the file input to get the list for your selected language — then edit it to suit your corpus before uploading. This is the recommended workflow for Hindi and Tamil, and for any domain-specific Bengali or English corpus.

If both the NLTK option and a custom file are active, the lists are automatically merged before modelling begins.

Language-Specific Guidance

Example Stopword File Format

Use plain text with one term per line. Encoding must be UTF-8.

word1
word2
word3

Parameters

The parameters you choose during upload affect how the model processes your texts and extracts topics. Some are intuitive, others more technical. Begin with recommended ranges — then adjust gradually based on the interpretability of your output.

Basic Parameters

Number of Topics
This controls how many thematic clusters the model will attempt to identify. It's not a fixed setting — it depends heavily on your corpus:

• Small, focused corpora (e.g., 30–50 short documents): try 5–10 topics
• Medium corpora (100–200 documents with some variation): try 10–20
• Large, diverse corpora (e.g., 500+ documents or full-length essays): you can try 30–50+, but interpret with care

Tip: If topics feel repetitive or unclear, reduce the number. If topics feel too broad, increase gradually. The goal is not coverage — it's clarity.

Language
Select the primary language of your uploaded documents. This controls which tokeniser, normaliser, and stopword list are used. Do not mix languages within a single run — upload multilingual corpora as separate analyses.

Normalisation
Reduces word forms to a common base. The available options depend on language:

• BN Stemming — reduces Bengali inflected forms via dictionary lookup (e.g. খাচ্ছে, খেয়েছে → খা). Recommended for most Bengali corpora.
• EN Lemmatisation — reduces English words to dictionary forms (e.g. running → run). Recommended.
• HI Lemmatisation — experimental Hindi lemmatisation via simplemma. Use cautiously.
• TA Stemming — dictionary-based Tamil stemming. Recommended for Tamil corpora.

Advanced Options

Remove Top Words (%)
Automatically discards the top N% most frequent words across the corpus. These often include generic verbs or connectors that don't add thematic value. Start with 5–10.

Filter: No Below
Removes words that appear in fewer than N documents. Useful to exclude rare noise — typos, one-off names, OCR artefacts. Recommended: 3–10.

Filter: No Above
Removes words that appear in more than this fraction of documents. These are often uninformative. Recommended: 0.5–0.7.

Alpha and Eta
These govern sparsity:

• Alpha: Lower values → fewer topics per document (sharper focus)
• Eta: Lower values → fewer words per topic (sharper boundaries)

Chunk Size
Controls how many documents are processed at once during training. This affects speed and memory use — not model quality. Default: 100–200.

Visualisations

Each visualisation in anvay helps you explore your topic model from a different angle: identifying overlap, strength, evolution, and prevalence.

Note: All visualisations display the top words for each topic on hover — making it easy to identify and interpret topics directly from the charts without switching back to the Report tab.

1. Topic Similarity Scatterplot

Closer dots = more similar topics. This chart plots topics based on their word distributions. Topics that are close together share many high-weight words. Use this to identify overlapping themes or cluster boundaries.

Axes: Both axes represent a two-dimensional projection of topic–word distributions. Distance between points reflects topic similarity; the axes themselves have no intrinsic semantic meaning.

2. Topic × Word Heatmap

Brighter cells = stronger topic–word link. Each cell shows how strongly a word is associated with a topic. Use this to explore overlap and word distribution depth. This view presents the same information shown in the bar chart, but emphasises patterns and contrast across topics.

Axes: The horizontal axis lists topic indices, and the vertical axis lists a globally selected set of high-weight words. Cell intensity reflects the strength of association between a word and a topic.

3. Topic Prevalence Pie Chart

Bigger slices = more dominant topics. This chart shows how much each topic contributes to the corpus overall.

Encoding: Slice size represents the cumulative weight of a topic across the corpus.

4. Top Word Frequency

High bars = frequent words in your corpus. Use this to inspect lexical dominance and identify potential noise.

Axes: The horizontal axis shows word tokens, and the vertical axis shows their frequency in the corpus after preprocessing.

5. Topic–Word Network Graph

Each topic node links to its top words. Blue circles are topics, grey ones are words. Helps visualise word overlap and outliers.

Encoding: Spatial layout reflects relational structure rather than scale; proximity and connections indicate association, not distance or magnitude.

6. Topic Evolution Over Time

Each line shows a topic's weight across time. Use this to detect historical shifts or evolving discourse patterns.

Axes: The horizontal axis represents document order (or a derived sequence such as time bins), and the vertical axis shows topic weight, tracing how topic presence changes across the corpus.

7. Topic Distribution Across Documents

Each dot shows a topic's weight in one document. Useful for detecting niche vs. broad topics and skewed presence.

Axes: The horizontal axis lists documents in the uploaded corpus, and the vertical axis shows topic weight, indicating how strongly each topic is represented in individual documents.

8. Topic Distance (Clustering Dendrogram)

Shorter branches = more similar topics. This dendrogram clusters topics by semantic similarity. Helps spot overlap or hierarchy.

Axes: The vertical axis shows inter-topic distance, while the horizontal arrangement reflects hierarchical grouping rather than a quantitative scale.

Report

The Report tab gives you a structured breakdown of the model's results — from high-level training stats to granular, paragraph-level illustrations. It's designed to support interpretive work, especially when working with complex or literary texts.

1. Training Overview

This section is designed to help you tune your topic model by reflecting on how the input data and training parameters shaped the model. It provides a snapshot of your corpus and the internal diagnostics logged during training. Used carefully, this is where interpretability begins.

Corpus Statistics

These values reflect the input that the model actually received, after all filtering and preprocessing. They help you assess whether your model is working with enough — and the right kind of — data.

• Total Documents: Each file uploaded is treated as one document. Fewer than 30 documents usually leads to unstable, overgeneralised topics. Over 100 allows more granularity.
• Average Document Length: If this is under 300 tokens, topics may feel thin or generic. In such cases, consider reducing stopword or rare word removal, disabling stemming, or merging short documents.
• Vocabulary Size: The number of unique tokens remaining after filtering. A range between 1,000–10,000 is often ideal for LDA.

Training Log

These logs come directly from the LDA engine (gensim), grouped into stages. They help you interpret model behaviour and stability.

• Token Filtering: Shows how many terms were discarded by your frequency thresholds.
• Training Configuration: Confirms what values were used for parameters like alpha, eta, and chunksize.
• Perplexity Estimates: Lower perplexity is usually better. Steady decline and flattening = good; rising after flattening = potential overtraining.
• Final Model Info: Marks the lifecycle stages of the model.

What to Do With This

• If your topics seem repetitive, generic, or incoherent, start by checking your average document length and vocabulary size.
• If training logs show rapid convergence but poor topics, try increasing passes or adjusting alpha.
• If perplexity never settles, it may indicate noisy input or a mismatch between the number of topics and corpus size.

↑ Back to top

2. Top Words per Topic

This section shows the top 10 words in each topic, ranked by their probability of occurring within that topic. These are taken directly from the model — no filtering, smoothing, or interpretation has been applied.

In technical terms, this is based on p(w | t) — the probability of a word given a topic. The higher the value, the more representative that word is for that topic. However, this measure does not account for whether the word is distinctive across topics.

How to Interpret

• Repetition across topics: If the same word appears in many topics, it may be common across the corpus — not distinctive. Consider adding it to your custom stopwords.
• Unusual terms: Some topics may include broken suffixes, particles, or OCR noise — signs of preprocessing issues, not model failure.
• Use your judgement: These word lists are prompts. Try to label each topic yourself — what theme or discourse might these words suggest?
• Check against documents: Return to the representative sentences. Do the top words actually appear in those contexts?

↑ Back to top

3. Topic Weights per Document

This section shows how each topic contributes to each document, based on the topic model's output. The weights are probabilistic, not categorical — every document is a mixture of topics, and every topic appears in multiple documents to varying degrees.

Each value represents p(t | d) — the model's estimate of how likely topic t is to occur in document d. The sum of all topic weights for a single document is 1.0.

How to Read This Table

• High values (e.g. ≥ 0.6): indicate a document where that topic is dominant.
• Flat distributions: If a document has many topics with small weights (e.g. 0.05–0.15), it may not fit cleanly into any one theme.
• Repeated patterns: If the same topic dominates across many documents, your model may be too coarse.
• Missing weights: A topic with 0.0 in a document does not appear there, or appears below the model's threshold.

Example: Suppose a document has the following topic weights:

Topic 2: 0.61, Topic 4: 0.24, Topic 7: 0.11

Topic 2 is strongly dominant; Topic 4 plays a secondary role; Topic 7 is minor but present.

↑ Back to top

4. Top Terms

These are the most frequently occurring words in the corpus after preprocessing. They are not tied to any specific topic but reflect recurring motifs and dominant vocabulary. If you see generic verbs, numbers, or formatting tokens here, consider adding them to your stopword list.

↑ Back to top

5. Topic Prevalence

Topic prevalence indicates how strongly each topic appears across the entire corpus. It is calculated by summing the topic's weight across all documents. Although topic weights do not have a fixed scale, a higher value generally means that the topic appears more consistently and with greater intensity in the dataset.

This table helps you identify which topics dominate the corpus overall — but note that very high values can sometimes reflect overfitting or topic redundancy. Use this view in combination with the document-level weights and top words to evaluate coverage and specificity.

↑ Back to top

6. Representative Sentences

For each topic, anvay displays one real sentence from the dataset where that topic is most strongly represented. These are not summaries — they are actual sentences taken from the original files.

Each sentence is accompanied by the filename and the topic's weight in that document. If no sentence strongly overlaps with the top words for that topic, it is marked as ⚠ Low Confidence.

"Topic 7: শ্রমিক, মজুরি, কারখানা → appears most clearly in a sentence describing a workers' protest speech. The texture is emotional, not abstract."

↑ Back to top

Walkthrough: Bengali News Corpus (150 Files)

This walkthrough explores how different filtering strategies shape the topics generated from a corpus of 150 Bengali news articles. Rather than presenting final outputs, it highlights how topic visibility is shaped by preprocessing decisions — and how different assumptions produce different readings.

Note: Because anvay uses a probabilistic model (LDA), your results may vary slightly even with the same corpus and parameters. This is expected behaviour and reflects the model's sensitivity to small variations in sampling.

Run A: Minimal Filtering (Latent Topics)

This configuration removes only the top 10% most frequent words and retains even the rarest tokens. The resulting topics are noisy but revealing — they surface underrepresented narrative fragments, scattered names, and infrastructural traces that typically disappear under stricter filtering.

Run B: Frequency Filtering (Dominant Topics)

This configuration applies both lower and upper frequency thresholds, removing extremely rare and overly common words. The resulting topics are more abstract, consistent, and legible — but many specific names and local contexts are lost in the process.

Named entities, places, and idiosyncratic phrasing are not incidental — they are structural elements of meaning. Standard filtering often erases these, making latent themes disappear.

What this walkthrough shows is that:
Legibility is a function of filtering. A topic doesn't appear because it's important — it appears because your filtering method allows it to survive.

What You Should Try

anvay defaults to a gentler filtering method (removing only the top 10% of words) so you can surface latent structure. But you can also enable frequency filtering if your goal is topical readability.

We recommend:

• Start with no no_below/no_above, just remove top 5–10% words
• Observe which topics emerge
• Then apply filters to examine what vanishes and what persists

This helps you understand your corpus — and your assumptions — much more than relying on a single run.

Closing Thought

Topic modelling is not neutral. The same corpus can yield very different "themes" depending on how you treat frequency. anvay is designed not to automate interpretation, but to reveal its conditions.

Limits & Sessions

anvay is a shared research tool hosted on a university server. To ensure fair and stable access for all users, it enforces limits on upload size, submission rate, and result availability. This page explains what those limits are and what to expect from each session.

Upload Limits

These limits are enforced before processing begins. If any file fails validation — wrong format, wrong encoding, too large — the entire submission is rejected and you will see a specific error message. No partial processing occurs.

Rate Limits

To prevent overload, anvay limits the number of submissions from a single IP address:

If you exceed these limits, you will see a 429 Too Many Requests error. Wait a moment and try again — the limit resets automatically. For research workflows that require many rapid iterations, we recommend preparing your corpus and parameters carefully before submitting, rather than making exploratory runs in quick succession.

Sessions and Result Availability

Every submission to anvay is independent. Each run generates a private, uniquely identified session — your results are not linked to any user account and are not visible to anyone else.

There is no login, no persistent storage, and no way to recover a session once it has expired. Plan accordingly — especially for large corpora where training may take several minutes.

Data and Privacy

anvay does not retain your files. Uploaded documents are stored only for the duration of processing, then deleted immediately — before the results page even loads. Result files are deleted after 2 hours. No content from your corpus is logged, stored, or shared.

anvay does not use cookies, tracking scripts, or third-party analytics. It does not display advertisements.